2024-01-21T07:15:54.538 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T07:15:54.538 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228 -- /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/orig.config.yaml 2024-01-21T07:15:54.589 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T07:15:54.681 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7524228' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8186 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuDCNfoxzHXB8fNTMhK4TGKKfDzG5B+jegV+VPwiryb1FXjUuBdz+DvMbp9Lxjl8omzQHsAGJ9DWURq/P/Iob0= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyDHvJhRVbhUnxp4R/WyGlaU29uLAD6NJJRgUKS/1Rrj8sF9tlF89hKMVyA6WhQ7wChR4oSjmz6o2egi5TjIpQ= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_23:17:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T07:15:54.681 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T07:15:54.682 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T07:15:54.682 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T07:15:54.684 INFO:teuthology.task.internal:Checking packages... 2024-01-21T07:15:54.707 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T07:15:54.707 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T07:15:54.707 INFO:teuthology.packaging:ref: None 2024-01-21T07:15:54.707 INFO:teuthology.packaging:tag: None 2024-01-21T07:15:54.707 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T07:15:54.707 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:15:54.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T07:15:54.857 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T07:15:54.858 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T07:15:54.869 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T07:15:54.869 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T07:15:54.886 INFO:teuthology.task.internal:Saving configuration 2024-01-21T07:15:54.906 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T07:15:54.917 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T07:15:54.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:10:40.288251', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuDCNfoxzHXB8fNTMhK4TGKKfDzG5B+jegV+VPwiryb1FXjUuBdz+DvMbp9Lxjl8omzQHsAGJ9DWURq/P/Iob0='} 2024-01-21T07:15:54.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:10:40.289985', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyDHvJhRVbhUnxp4R/WyGlaU29uLAD6NJJRgUKS/1Rrj8sF9tlF89hKMVyA6WhQ7wChR4oSjmz6o2egi5TjIpQ='} 2024-01-21T07:15:54.958 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T07:15:54.965 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T07:15:54.965 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T07:15:54.965 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T07:15:55.033 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efe4480b160>, signals=[15]) 2024-01-21T07:15:55.034 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T07:15:55.043 INFO:teuthology.task.internal:Opening connections... 2024-01-21T07:15:55.043 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-01-21T07:15:55.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:15:55.123 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-01-21T07:15:55.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:15:55.195 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T07:15:55.204 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-01-21T07:15:55.226 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-01-21T07:15:55.226 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-01-21T07:15:55.287 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T07:15:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-01-21T07:15:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T07:15:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T07:15:55.288 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T07:15:55.288 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-01-21T07:15:55.308 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-01-21T07:15:55.330 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-01-21T07:15:55.331 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-01-21T07:15:55.392 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T07:15:55.393 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-01-21T07:15:55.393 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T07:15:55.393 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T07:15:55.393 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T07:15:55.393 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-01-21T07:15:55.408 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T07:15:55.424 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T07:15:55.435 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T07:15:55.435 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-01-21T07:15:55.437 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-01-21T07:15:55.458 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T07:15:55.472 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T07:15:55.472 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-01-21T07:15:55.499 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-01-21T07:15:55.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T07:15:55.584 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T07:15:55.604 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T07:15:55.604 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T07:15:55.604 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T07:15:55.605 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T07:15:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:15:55.625 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-01-21T07:15:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:15:55.626 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-01-21T07:15:55.687 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-489.el8.x86_64 2024-01-21T07:15:55.687 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-489.el8.x86_64 2024-01-21T07:15:55.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-01-21T07:15:55.688 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-489.el8.x86_64 2024-01-21T07:15:55.688 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-489.el8.x86_64 2024-01-21T07:15:55.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-01-21T07:15:59.625 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 18 MB/s | 26 MB 00:01 2024-01-21T07:16:02.149 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 6.9 MB/s | 26 MB 00:03 2024-01-21T07:16:05.483 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 8.8 MB/s | 10 MB 00:01 2024-01-21T07:16:07.633 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-01-21T07:16:07.807 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-01-21T07:16:08.318 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-01-21T07:16:08.536 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-21T07:16:09.791 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-01-21T07:16:10.289 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2024-01-21T07:16:10.366 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2024-01-21T07:16:10.601 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-21T07:16:11.854 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-01-21T07:16:14.036 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 653 kB/s | 24 kB 00:00 2024-01-21T07:16:15.662 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 416 kB/s | 24 kB 00:00 2024-01-21T07:16:16.933 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T07:16:16.933 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T07:16:17.039 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T07:16:17.040 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 89 M 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-01-21T07:16:17.041 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:16:18.112 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 9.9 MB/s | 10 MB 00:01 2024-01-21T07:16:18.542 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T07:16:18.542 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T07:16:18.647 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:16:18.647 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:16:18.647 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout:Install 3 Packages 2024-01-21T07:16:18.648 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:18.649 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 89 M 2024-01-21T07:16:18.649 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 96 M 2024-01-21T07:16:18.649 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:16:19.004 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-01-21T07:16:19.521 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 18 MB/s | 43 MB 00:02 2024-01-21T07:16:19.521 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:16:19.521 INFO:teuthology.orchestra.run.smithi107.stdout:Total 36 MB/s | 89 MB 00:02 2024-01-21T07:16:19.705 INFO:teuthology.orchestra.run.smithi115.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-01-21T07:16:20.198 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:16:20.635 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:16:20.635 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:16:20.663 INFO:teuthology.orchestra.run.smithi115.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 18 MB/s | 36 MB 00:01 2024-01-21T07:16:21.172 INFO:teuthology.orchestra.run.smithi115.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-01-21T07:16:21.172 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:16:21.172 INFO:teuthology.orchestra.run.smithi115.stdout:Total 35 MB/s | 89 MB 00:02 2024-01-21T07:16:21.849 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:16:21.886 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:16:21.907 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:16:22.262 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:16:22.262 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:16:23.493 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:16:23.517 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:16:24.637 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:16:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:26.242 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:16:26.711 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:28.749 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:29.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:30.311 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:30.743 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:32.605 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:32.905 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:34.160 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:34.423 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:54.880 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:54.903 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:55.244 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:55.244 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:55.244 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:55.769 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:16:55.910 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:16:56.168 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:16:56.168 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:16:56.168 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T07:16:56.168 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T07:16:56.168 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T07:16:56.168 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T07:16:56.169 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T07:16:56.169 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:16:56.358 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:56.377 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:16:56.377 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:16:56.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-01-21T07:16:56.381 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 129.el8 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-01-21T07:16:56.410 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9265586 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:56.411 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T07:16:56.412 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T07:16:56.412 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T07:16:56.412 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-01-21T07:16:56.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:16:56.412 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T07:16:56.413 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T07:16:56.413 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-535.el8.x86_64 2024-01-21T07:16:56.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T07:16:56.726 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:56.726 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T07:16:56.726 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:57.193 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:16:57.337 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:16:57.552 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:16:57.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:16:57.552 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T07:16:57.552 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T07:16:57.552 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T07:16:57.552 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T07:16:57.553 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T07:16:57.553 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T07:16:57.760 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T07:16:57.761 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T07:16:57.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.02 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 129.el8 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Group : System Environment/Base 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 9265586 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T07:16:57.794 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Relocations : (not relocatable) 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : CentOS BuildSystem 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:16:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T07:16:57.796 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T07:16:57.796 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 4.18.0-535.el8.x86_64 2024-01-21T07:16:57.797 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T07:16:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-01-21T07:16:59.487 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-01-21T07:17:00.226 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-01-21T07:17:00.228 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-01-21T07:17:00.262 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.zF9OSO8xRq 2024-01-21T07:17:00.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zF9OSO8xRq 2024-01-21T07:17:00.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.zF9OSO8xRq 2024-01-21T07:17:00.443 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.zF9OSO8xRq is 6KB 2024-01-21T07:17:00.492 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.zF9OSO8xRq 2024-01-21T07:17:00.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T07:17:00.575 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T07:17:00.575 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T07:17:00.576 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T07:17:00.576 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T07:17:00.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T07:17:00.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-01-21T07:17:01.654 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-01-21T07:17:01.655 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-01-21T07:17:01.697 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.YXoofmtu8G 2024-01-21T07:17:01.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YXoofmtu8G 2024-01-21T07:17:01.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.YXoofmtu8G 2024-01-21T07:17:01.911 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.YXoofmtu8G is 6KB 2024-01-21T07:17:01.959 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.YXoofmtu8G 2024-01-21T07:17:01.976 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T07:17:02.052 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T07:17:02.052 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T07:17:02.052 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T07:17:02.052 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T07:17:02.053 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T07:17:02.197 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-01-21T07:17:30.755 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T07:17:30.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:17:30.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:17:32.200 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T07:17:32.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:17:32.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:17:49.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-01-21T07:17:50.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-01-21T07:17:58.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:17:58.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:17:59.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:17:59.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:18:01.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-01-21T07:18:02.592 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-01-21T07:18:13.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:18:13.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:18:14.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:18:14.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:18:31.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-01-21T07:18:33.024 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-01-21T07:18:46.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:18:46.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:18:47.133 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-01-21T07:18:47.365 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:18:47.365 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T07:18:47.365 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-01-21T07:18:47.382 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-535.el8.x86_64 2024-01-21T07:18:47.382 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T07:18:47.382 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T07:18:47.382 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T07:18:48.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:18:48.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:18:48.384 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T07:18:48.384 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T07:18:48.404 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-01-21T07:18:48.455 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-01-21T07:18:48.467 DEBUG:teuthology.parallel:result is None 2024-01-21T07:18:48.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:18:48.769 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T07:18:48.769 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-01-21T07:18:48.786 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-535.el8.x86_64 2024-01-21T07:18:48.786 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T07:18:48.786 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T07:18:48.786 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T07:18:49.787 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T07:18:49.788 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T07:18:49.873 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-01-21T07:18:49.887 DEBUG:teuthology.parallel:result is None 2024-01-21T07:18:49.887 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T07:18:49.896 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T07:18:49.896 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T07:18:49.899 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T07:18:49.916 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T07:18:49.943 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T07:18:50.002 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T07:18:50.002 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T07:18:50.004 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T07:18:50.027 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T07:18:50.034 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T07:18:50.034 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T07:18:50.059 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T07:18:50.098 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:18:50.108 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:18:50.117 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:18:50.127 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T07:18:50.128 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T07:18:50.135 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T07:18:50.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T07:18:50.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T07:18:50.176 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T07:18:50.195 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T07:18:50.195 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T07:18:50.212 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T07:18:50.232 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:18:50.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:18:50.346 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:18:50.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:18:50.466 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:18:50.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T07:18:50.530 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:18:50.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T07:18:50.628 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:18:50.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T07:18:50.756 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:18:50.756 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T07:18:50.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-01-21T07:18:50.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-01-21T07:18:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:18:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:18:51.324 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T07:18:51.332 INFO:teuthology.task.internal:Starting timer... 2024-01-21T07:18:51.332 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T07:18:51.397 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T07:18:51.446 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T07:18:51.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-01-21T07:18:51.643 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-01-21T07:18:51.645 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-01-21T07:18:51.795 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-01-21T07:18:51.823 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T07:18:51.823 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-01-21T07:18:51.882 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-01-21T07:18:51.882 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-01-21T07:18:52.455 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-01-21T07:18:52.455 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2024-01-21T07:18:52.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:18:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:18:53.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:18:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:18:53.200 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T07:18:53.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-01-21T07:18:53.666 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-01-21T07:18:53.812 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T07:18:53.823 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-21T07:18:53.828 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-21T07:18:53.938 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T07:18:53.952 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T07:18:53.953 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T07:24:29.341 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2024-01-21T07:24:29.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:24:29.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:24:29.432 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-01-21T07:24:29.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-01-21T07:24:29.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:24:29.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T07:24:29.581 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-01-21T07:24:29.652 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-01-21T07:24:29.705 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T07:24:29.715 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T07:24:29.716 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T07:24:29.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T07:24:29.718 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T07:24:29.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T07:24:29.750 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T07:24:29.751 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T07:24:29.764 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T07:24:29.765 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T07:24:29.825 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-01-21T07:24:29.836 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-01-21T07:24:29.838 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-01-21T07:24:29.849 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-01-21T07:24:29.850 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T07:24:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T07:24:29.863 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T07:24:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T07:24:29.925 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2024-01-21T07:24:29.927 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T07:24:29.927 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-01-21T07:24:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.928 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.954 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.958 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:24:29.959 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T07:24:29.968 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T07:24:29.969 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2024-01-21T07:24:29.969 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T07:24:29.969 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2024-01-21T07:24:29.970 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T07:24:30.722 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 07:23:09 AM UTC. 2024-01-21T07:24:30.754 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 07:23:09 AM UTC. 2024-01-21T07:24:32.644 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout:Resetting modules: 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:32.645 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:Resetting modules: 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:32.652 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:32.906 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:24:32.912 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:24:33.717 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:23:09 AM UTC. 2024-01-21T07:24:33.717 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:23:09 AM UTC. 2024-01-21T07:24:35.643 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:24:35.671 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:24:35.677 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout:Installing group/module packages: 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T07:24:35.678 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:Installing module profiles: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools/common 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling module streams: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools rhel8 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:24:35.679 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-01-21T07:24:35.680 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:35.680 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 27 M 2024-01-21T07:24:35.680 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 89 M 2024-01-21T07:24:35.680 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout:Installing group/module packages: 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T07:24:35.706 INFO:teuthology.orchestra.run.smithi115.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout:Installing module profiles: 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools/common 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2024-01-21T07:24:35.707 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools rhel8 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:35.708 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 27 M 2024-01-21T07:24:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 89 M 2024-01-21T07:24:35.709 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:24:36.014 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.5 MB/s | 750 kB 00:00 2024-01-21T07:24:36.049 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.1 MB/s | 1.0 MB 00:00 2024-01-21T07:24:36.075 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 620 kB/s | 38 kB 00:00 2024-01-21T07:24:36.108 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2024-01-21T07:24:36.173 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2024-01-21T07:24:36.175 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2024-01-21T07:24:36.199 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.4 MB/s | 1.0 MB 00:00 2024-01-21T07:24:36.232 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2024-01-21T07:24:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2024-01-21T07:24:36.276 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 20 MB/s | 3.9 MB 00:00 2024-01-21T07:24:36.292 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2024-01-21T07:24:36.341 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2024-01-21T07:24:36.418 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 14 MB/s | 9.6 MB 00:00 2024-01-21T07:24:36.435 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 591 kB/s | 94 kB 00:00 2024-01-21T07:24:36.475 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 17 MB/s | 9.6 MB 00:00 2024-01-21T07:24:36.485 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2024-01-21T07:24:36.491 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 165 kB/s | 25 kB 00:00 2024-01-21T07:24:36.535 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 22 MB/s | 2.5 MB 00:00 2024-01-21T07:24:36.552 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 612 kB/s | 41 kB 00:00 2024-01-21T07:24:36.558 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2024-01-21T07:24:36.584 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 864 kB/s | 94 kB 00:00 2024-01-21T07:24:36.667 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 23 MB/s | 2.5 MB 00:00 2024-01-21T07:24:36.693 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 484 kB/s | 53 kB 00:00 2024-01-21T07:24:36.702 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 20 MB/s | 8.1 MB 00:00 2024-01-21T07:24:36.703 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:24:36.703 INFO:teuthology.orchestra.run.smithi107.stdout:Total 26 MB/s | 27 MB 00:01 2024-01-21T07:24:36.718 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 814 kB/s | 41 kB 00:00 2024-01-21T07:24:36.877 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2024-01-21T07:24:36.877 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:24:36.877 INFO:teuthology.orchestra.run.smithi115.stdout:Total 23 MB/s | 27 MB 00:01 2024-01-21T07:24:36.968 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:24:36.989 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:24:36.989 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:24:37.141 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:24:37.162 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:24:37.162 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:24:37.245 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:24:37.251 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:24:37.420 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:24:37.423 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:24:37.943 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:24:38.152 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:24:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T07:24:38.366 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T07:24:38.391 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T07:24:38.552 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T07:24:38.623 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T07:24:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T07:24:38.879 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T07:24:38.937 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T07:24:39.457 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T07:24:39.473 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T07:24:40.410 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T07:24:40.511 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T07:24:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T07:24:40.765 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T07:24:40.884 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T07:24:40.998 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T07:24:41.173 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T07:24:41.296 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T07:24:41.579 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T07:24:41.712 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T07:24:42.468 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T07:24:42.633 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T07:24:42.682 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T07:24:42.856 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T07:24:42.867 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T07:24:43.074 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T07:24:43.305 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T07:24:43.306 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T07:24:43.481 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T07:24:43.482 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T07:24:43.482 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T07:24:43.482 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:43.841 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:24:43.842 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:24:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T07:24:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T07:24:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T07:24:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:24:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:24:44.126 DEBUG:teuthology.parallel:result is None 2024-01-21T07:24:44.330 DEBUG:teuthology.parallel:result is None 2024-01-21T07:24:44.330 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T07:24:44.338 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T07:24:44.338 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2024-01-21T07:24:44.339 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T07:24:44.339 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2024-01-21T07:24:44.339 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T07:24:44.544 DEBUG:teuthology.parallel:result is None 2024-01-21T07:24:44.570 DEBUG:teuthology.parallel:result is None 2024-01-21T07:24:44.570 INFO:teuthology.run_tasks:Running task install... 2024-01-21T07:24:44.579 DEBUG:teuthology.task.install:project ceph 2024-01-21T07:24:44.579 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T07:24:44.579 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T07:24:44.579 INFO:teuthology.task.install:Using flavor: default 2024-01-21T07:24:44.587 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T07:24:44.587 INFO:teuthology.task.install:extra packages: [] 2024-01-21T07:24:44.587 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T07:24:44.588 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:24:44.588 INFO:teuthology.packaging:ref: None 2024-01-21T07:24:44.588 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:24:44.588 INFO:teuthology.packaging:branch: None 2024-01-21T07:24:44.588 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:24:44.784 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T07:24:44.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T07:24:44.787 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T07:24:44.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:24:44.787 INFO:teuthology.packaging:ref: None 2024-01-21T07:24:44.787 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:24:44.787 INFO:teuthology.packaging:branch: None 2024-01-21T07:24:44.787 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:24:44.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T07:24:44.929 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T07:24:44.929 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T07:24:44.967 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T07:24:44.967 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:24:44.967 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T07:24:44.995 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T07:24:44.996 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:24:44.996 INFO:teuthology.packaging:ref: None 2024-01-21T07:24:44.996 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:24:44.996 INFO:teuthology.packaging:branch: None 2024-01-21T07:24:44.996 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:24:44.996 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T07:24:45.002 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T07:24:45.002 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T07:24:45.025 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T07:24:45.025 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:24:45.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T07:24:45.054 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T07:24:45.054 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:24:45.054 INFO:teuthology.packaging:ref: None 2024-01-21T07:24:45.054 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:24:45.054 INFO:teuthology.packaging:branch: None 2024-01-21T07:24:45.055 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:24:45.055 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T07:24:45.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T07:24:45.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T07:24:45.140 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T07:24:45.199 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T07:24:45.205 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-01-21T07:24:45.207 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-01-21T07:24:45.266 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-01-21T07:24:45.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-01-21T07:24:45.469 INFO:teuthology.orchestra.run.smithi115.stdout:76 files removed 2024-01-21T07:24:45.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2024-01-21T07:24:45.530 INFO:teuthology.orchestra.run.smithi107.stdout:76 files removed 2024-01-21T07:24:45.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-01-21T07:24:46.034 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 555 kB/s | 74 kB 00:00 2024-01-21T07:24:46.051 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 828 kB/s | 74 kB 00:00 2024-01-21T07:24:46.177 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 172 kB/s | 15 kB 00:00 2024-01-21T07:24:46.193 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 126 kB/s | 15 kB 00:00 2024-01-21T07:24:46.327 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 16 kB/s | 1.7 kB 00:00 2024-01-21T07:24:46.344 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 13 kB/s | 1.7 kB 00:00 2024-01-21T07:24:47.377 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 26 MB/s | 26 MB 00:01 2024-01-21T07:24:47.445 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 25 MB/s | 26 MB 00:01 2024-01-21T07:24:53.121 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-01-21T07:24:53.261 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 8.5 MB/s | 10 MB 00:01 2024-01-21T07:24:55.271 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-01-21T07:24:55.395 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-01-21T07:24:57.888 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 2.8 kB/s | 6.9 kB 00:02 2024-01-21T07:24:57.970 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 2.8 kB/s | 6.9 kB 00:02 2024-01-21T07:24:58.497 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - PowerTools 9.2 MB/s | 4.7 MB 00:00 2024-01-21T07:24:58.688 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - PowerTools 7.6 MB/s | 4.7 MB 00:00 2024-01-21T07:24:59.624 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-01-21T07:24:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-01-21T07:25:01.568 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 16 MB 00:01 2024-01-21T07:25:05.491 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 241 kB/s | 24 kB 00:00 2024-01-21T07:25:08.880 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 16 MB 00:09 2024-01-21T07:25:08.945 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:25:08.948 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:08.948 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T07:25:08.949 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T07:25:08.950 INFO:teuthology.orchestra.run.smithi107.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout:Install 25 Packages 2024-01-21T07:25:08.951 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-01-21T07:25:08.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:08.952 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 59 M 2024-01-21T07:25:08.952 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:25:09.233 INFO:teuthology.orchestra.run.smithi107.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2024-01-21T07:25:09.267 INFO:teuthology.orchestra.run.smithi107.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 748 kB/s | 25 kB 00:00 2024-01-21T07:25:09.300 INFO:teuthology.orchestra.run.smithi107.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T07:25:09.334 INFO:teuthology.orchestra.run.smithi107.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T07:25:09.451 INFO:teuthology.orchestra.run.smithi107.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2024-01-21T07:25:09.476 INFO:teuthology.orchestra.run.smithi107.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T07:25:09.501 INFO:teuthology.orchestra.run.smithi107.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T07:25:09.527 INFO:teuthology.orchestra.run.smithi107.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-21T07:25:09.561 INFO:teuthology.orchestra.run.smithi107.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T07:25:09.611 INFO:teuthology.orchestra.run.smithi107.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.2 MB/s | 368 kB 00:00 2024-01-21T07:25:09.636 INFO:teuthology.orchestra.run.smithi107.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T07:25:09.797 INFO:teuthology.orchestra.run.smithi107.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-01-21T07:25:10.078 INFO:teuthology.orchestra.run.smithi107.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-21T07:25:10.362 INFO:teuthology.orchestra.run.smithi107.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 200 kB/s | 145 kB 00:00 2024-01-21T07:25:10.560 INFO:teuthology.orchestra.run.smithi107.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 223 kB/s | 44 kB 00:00 2024-01-21T07:25:10.596 INFO:teuthology.orchestra.run.smithi107.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 847 kB/s | 677 kB 00:00 2024-01-21T07:25:10.621 INFO:teuthology.orchestra.run.smithi107.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 478 kB/s | 259 kB 00:00 2024-01-21T07:25:10.872 INFO:teuthology.orchestra.run.smithi107.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 135 kB/s | 42 kB 00:00 2024-01-21T07:25:10.920 INFO:teuthology.orchestra.run.smithi107.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 145 kB/s | 47 kB 00:00 2024-01-21T07:25:10.947 INFO:teuthology.orchestra.run.smithi107.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 246 kB/s | 80 kB 00:00 2024-01-21T07:25:11.389 INFO:teuthology.orchestra.run.smithi107.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 225 kB/s | 116 kB 00:00 2024-01-21T07:25:11.456 INFO:teuthology.orchestra.run.smithi107.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 356 kB/s | 181 kB 00:00 2024-01-21T07:25:11.498 INFO:teuthology.orchestra.run.smithi107.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 530 kB/s | 306 kB 00:00 2024-01-21T07:25:11.523 INFO:teuthology.orchestra.run.smithi107.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 440 kB/s | 59 kB 00:00 2024-01-21T07:25:11.573 INFO:teuthology.orchestra.run.smithi107.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 640 kB/s | 75 kB 00:00 2024-01-21T07:25:11.650 INFO:teuthology.orchestra.run.smithi107.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-01-21T07:25:11.716 INFO:teuthology.orchestra.run.smithi107.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-01-21T07:25:11.720 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:25:11.721 INFO:teuthology.orchestra.run.smithi107.stdout:Total 21 MB/s | 59 MB 00:02 2024-01-21T07:25:11.791 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:25:11.839 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:25:11.839 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:25:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:25:12.201 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:25:12.737 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 287 kB/s | 24 kB 00:00 2024-01-21T07:25:13.343 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:25:13.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T07:25:13.797 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T07:25:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T07:25:14.371 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T07:25:14.584 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:14.691 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:14.944 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T07:25:15.092 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:15.152 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:15.402 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T07:25:15.943 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T07:25:16.184 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:16.259 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:16.276 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:25:16.279 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:16.279 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:25:16.279 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T07:25:16.280 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T07:25:16.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout:Install 25 Packages 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2024-01-21T07:25:16.282 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:16.283 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 59 M 2024-01-21T07:25:16.283 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:25:16.464 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T07:25:16.634 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T07:25:16.681 INFO:teuthology.orchestra.run.smithi115.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2024-01-21T07:25:16.683 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T07:25:16.698 INFO:teuthology.orchestra.run.smithi115.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-21T07:25:16.731 INFO:teuthology.orchestra.run.smithi115.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-01-21T07:25:16.782 INFO:teuthology.orchestra.run.smithi115.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 9.5 MB/s | 486 kB 00:00 2024-01-21T07:25:16.874 INFO:teuthology.orchestra.run.smithi115.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-01-21T07:25:16.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T07:25:16.899 INFO:teuthology.orchestra.run.smithi115.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-21T07:25:16.924 INFO:teuthology.orchestra.run.smithi115.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T07:25:16.950 INFO:teuthology.orchestra.run.smithi115.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-21T07:25:16.992 INFO:teuthology.orchestra.run.smithi115.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2024-01-21T07:25:17.025 INFO:teuthology.orchestra.run.smithi115.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T07:25:17.051 INFO:teuthology.orchestra.run.smithi115.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T07:25:17.068 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T07:25:17.185 INFO:teuthology.orchestra.run.smithi115.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-01-21T07:25:17.250 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T07:25:17.327 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T07:25:17.451 INFO:teuthology.orchestra.run.smithi115.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-21T07:25:17.477 INFO:teuthology.orchestra.run.smithi115.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 341 kB/s | 145 kB 00:00 2024-01-21T07:25:17.529 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:17.535 INFO:teuthology.orchestra.run.smithi115.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 751 kB/s | 44 kB 00:00 2024-01-21T07:25:17.572 INFO:teuthology.orchestra.run.smithi115.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-01-21T07:25:17.652 INFO:teuthology.orchestra.run.smithi115.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2024-01-21T07:25:17.844 INFO:teuthology.orchestra.run.smithi115.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 137 kB/s | 42 kB 00:00 2024-01-21T07:25:17.878 INFO:teuthology.orchestra.run.smithi115.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 152 kB/s | 47 kB 00:00 2024-01-21T07:25:17.953 INFO:teuthology.orchestra.run.smithi115.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 267 kB/s | 80 kB 00:00 2024-01-21T07:25:17.960 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:18.117 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T07:25:18.215 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T07:25:18.412 INFO:teuthology.orchestra.run.smithi115.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 205 kB/s | 116 kB 00:00 2024-01-21T07:25:18.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T07:25:18.529 INFO:teuthology.orchestra.run.smithi115.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 471 kB/s | 306 kB 00:00 2024-01-21T07:25:18.562 INFO:teuthology.orchestra.run.smithi115.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 299 kB/s | 181 kB 00:00 2024-01-21T07:25:18.588 INFO:teuthology.orchestra.run.smithi115.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 335 kB/s | 59 kB 00:00 2024-01-21T07:25:18.668 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T07:25:18.671 INFO:teuthology.orchestra.run.smithi115.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-01-21T07:25:18.697 INFO:teuthology.orchestra.run.smithi115.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 448 kB/s | 75 kB 00:00 2024-01-21T07:25:18.755 INFO:teuthology.orchestra.run.smithi115.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-21T07:25:18.760 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:25:18.760 INFO:teuthology.orchestra.run.smithi115.stdout:Total 24 MB/s | 59 MB 00:02 2024-01-21T07:25:18.808 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T07:25:18.827 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:25:18.875 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:25:18.875 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:25:18.886 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T07:25:19.100 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T07:25:19.226 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:25:19.229 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:25:19.275 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T07:25:19.490 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T07:25:19.725 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T07:25:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:25:20.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T07:25:20.879 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T07:25:21.043 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T07:25:21.443 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T07:25:21.618 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:21.722 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:21.984 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T07:25:22.143 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:22.209 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:22.245 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:22.458 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:22.469 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T07:25:23.001 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T07:25:23.143 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:23.172 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:23.206 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:23.389 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T07:25:23.489 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T07:25:23.576 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T07:25:23.594 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T07:25:23.631 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T07:25:23.749 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:23.791 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T07:25:24.001 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T07:25:24.184 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T07:25:24.258 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T07:25:24.443 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:24.876 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:25.259 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T07:25:25.361 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T07:25:25.580 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T07:25:25.787 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T07:25:25.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T07:25:26.028 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T07:25:26.219 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T07:25:26.384 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T07:25:26.603 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T07:25:26.859 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T07:25:29.442 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:29.750 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:30.461 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T07:25:31.123 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T07:25:31.208 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T07:25:31.316 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /run 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2024-01-21T07:25:42.693 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:44.051 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T07:25:44.113 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T07:25:44.113 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T07:25:44.363 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T07:25:44.363 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T07:25:45.067 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T07:25:45.068 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T07:25:45.069 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T07:25:45.069 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T07:25:45.069 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:45.069 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T07:25:45.069 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T07:25:45.955 INFO:teuthology.orchestra.run.smithi107.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.956 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:45.957 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:25:46.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-01-21T07:25:47.001 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:41 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:25:48.482 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:25:48.483 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:48.483 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-01-21T07:25:48.484 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:48.485 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 53 M 2024-01-21T07:25:48.485 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 229 M 2024-01-21T07:25:48.485 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:25:48.646 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.4 MB/s | 169 kB 00:00 2024-01-21T07:25:48.729 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-01-21T07:25:48.821 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2024-01-21T07:25:48.846 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 940 kB/s | 188 kB 00:00 2024-01-21T07:25:49.313 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-01-21T07:25:49.363 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 130 kB/s | 71 kB 00:00 2024-01-21T07:25:49.365 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:25:49.366 INFO:teuthology.orchestra.run.smithi107.stdout:Total 60 MB/s | 53 MB 00:00 2024-01-21T07:25:49.408 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:25:49.429 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:25:49.429 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:25:49.748 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:25:49.751 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:25:50.253 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:25:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:50.370 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-01-21T07:25:50.370 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-01-21T07:25:50.370 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2024-01-21T07:25:50.370 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2024-01-21T07:25:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:50.520 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T07:25:50.716 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T07:25:50.884 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T07:25:50.950 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T07:25:51.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T07:25:51.318 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T07:25:51.701 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T07:25:51.761 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T07:25:51.761 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T07:25:51.980 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T07:25:51.980 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:52.642 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:52.642 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T07:25:52.643 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T07:25:52.644 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:25:53.585 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.586 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:53.587 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:25:53.854 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2024-01-21T07:25:54.582 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:25:56.067 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T07:25:56.068 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 53 M 2024-01-21T07:25:56.069 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 229 M 2024-01-21T07:25:56.070 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:25:56.152 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T07:25:56.167 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 169 kB 00:00 2024-01-21T07:25:56.343 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 874 kB/s | 203 kB 00:00 2024-01-21T07:25:56.368 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 939 kB/s | 188 kB 00:00 2024-01-21T07:25:56.415 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T07:25:56.426 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-01-21T07:25:56.452 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-01-21T07:25:56.705 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T07:25:56.705 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T07:25:56.705 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T07:25:56.705 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T07:25:56.705 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T07:25:56.706 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T07:25:56.919 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2024-01-21T07:25:56.921 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:25:56.921 INFO:teuthology.orchestra.run.smithi115.stdout:Total 62 MB/s | 53 MB 00:00 2024-01-21T07:25:56.963 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:25:56.984 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:25:56.984 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:57.139 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:25:57.296 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:25:57.299 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:25:57.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-01-21T07:25:57.776 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:25:57.979 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T07:25:58.103 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:25:58.173 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T07:25:58.302 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T07:25:58.382 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T07:25:58.575 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T07:25:58.718 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T07:25:59.587 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T07:25:59.590 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T07:25:59.591 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T07:25:59.592 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:Install 32 Packages 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 31 M 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 111 M 2024-01-21T07:25:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:00.281 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 103 kB/s | 6.3 kB 00:00 2024-01-21T07:26:00.331 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-01-21T07:26:00.389 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-21T07:26:00.482 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2024-01-21T07:26:00.507 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 1.8 MB/s | 220 kB 00:00 2024-01-21T07:26:00.741 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T07:26:00.757 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-01-21T07:26:00.783 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 522 kB/s | 157 kB 00:00 2024-01-21T07:26:00.833 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 758 kB/s | 57 kB 00:00 2024-01-21T07:26:00.900 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-01-21T07:26:00.925 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 562 kB/s | 103 kB 00:00 2024-01-21T07:26:01.329 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 502 kB/s | 249 kB 00:00 2024-01-21T07:26:01.359 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 251 kB/s | 109 kB 00:00 2024-01-21T07:26:01.501 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-01-21T07:26:01.701 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 117 kB/s | 44 kB 00:00 2024-01-21T07:26:01.793 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 427 kB/s | 185 kB 00:00 2024-01-21T07:26:01.818 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 544 kB/s | 173 kB 00:00 2024-01-21T07:26:01.910 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-01-21T07:26:01.936 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-01-21T07:26:02.019 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2024-01-21T07:26:02.069 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 440 kB/s | 59 kB 00:00 2024-01-21T07:26:02.103 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 958 kB/s | 384 kB 00:00 2024-01-21T07:26:02.187 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-01-21T07:26:02.212 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-01-21T07:26:02.237 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-01-21T07:26:02.312 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 190 kB/s | 24 kB 00:00 2024-01-21T07:26:02.338 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-01-21T07:26:02.363 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-01-21T07:26:02.480 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 732 kB/s | 85 kB 00:00 2024-01-21T07:26:02.505 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.3 MB/s | 255 kB 00:00 2024-01-21T07:26:02.530 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-01-21T07:26:02.622 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 162 kB/s | 23 kB 00:00 2024-01-21T07:26:02.624 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:02.624 INFO:teuthology.orchestra.run.smithi107.stdout:Total 10 MB/s | 31 MB 00:03 2024-01-21T07:26:02.704 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:02.743 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:02.743 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:03.067 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:03.070 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:03.558 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T07:26:03.799 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T07:26:04.087 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T07:26:04.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T07:26:04.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T07:26:04.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T07:26:04.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T07:26:04.088 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T07:26:04.236 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:04.436 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T07:26:04.495 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T07:26:04.495 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:04.496 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:04.689 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T07:26:04.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2024-01-21T07:26:04.930 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T07:26:05.171 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T07:26:05.369 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T07:26:05.450 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T07:26:05.743 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T07:26:05.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T07:26:06.136 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T07:26:06.333 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T07:26:06.596 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T07:26:06.862 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T07:26:06.933 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:06.936 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T07:26:06.937 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T07:26:06.938 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout:Install 32 Packages 2024-01-21T07:26:06.939 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:06.940 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 31 M 2024-01-21T07:26:06.940 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 111 M 2024-01-21T07:26:06.940 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:07.220 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T07:26:07.277 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 127 kB/s | 6.3 kB 00:00 2024-01-21T07:26:07.327 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-01-21T07:26:07.378 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-01-21T07:26:07.411 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-21T07:26:07.475 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T07:26:07.487 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-01-21T07:26:07.663 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T07:26:07.745 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T07:26:07.779 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2024-01-21T07:26:07.804 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 399 kB/s | 157 kB 00:00 2024-01-21T07:26:07.863 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 682 kB/s | 57 kB 00:00 2024-01-21T07:26:07.913 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.1 MB/s | 457 kB 00:00 2024-01-21T07:26:07.920 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T07:26:07.947 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 514 kB/s | 103 kB 00:00 2024-01-21T07:26:08.081 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T07:26:08.305 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 304 kB/s | 109 kB 00:00 2024-01-21T07:26:08.331 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 532 kB/s | 249 kB 00:00 2024-01-21T07:26:08.405 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T07:26:08.448 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.2 MB/s | 641 kB 00:00 2024-01-21T07:26:08.473 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2024-01-21T07:26:08.498 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2024-01-21T07:26:08.549 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-01-21T07:26:08.574 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.7 MB/s | 384 kB 00:00 2024-01-21T07:26:08.624 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-01-21T07:26:08.649 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 858 kB/s | 173 kB 00:00 2024-01-21T07:26:08.661 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T07:26:08.675 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2024-01-21T07:26:08.700 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 780 kB/s | 59 kB 00:00 2024-01-21T07:26:08.725 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-01-21T07:26:08.750 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2024-01-21T07:26:08.776 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-01-21T07:26:08.801 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 318 kB/s | 24 kB 00:00 2024-01-21T07:26:08.826 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-01-21T07:26:08.833 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T07:26:08.851 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 366 kB/s | 27 kB 00:00 2024-01-21T07:26:08.868 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-01-21T07:26:08.894 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-01-21T07:26:08.919 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2024-01-21T07:26:08.944 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 927 kB/s | 85 kB 00:00 2024-01-21T07:26:08.945 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:08.945 INFO:teuthology.orchestra.run.smithi115.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-21T07:26:09.026 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:09.064 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:09.064 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:09.067 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T07:26:09.306 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T07:26:09.382 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:09.386 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:09.551 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T07:26:09.794 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T07:26:10.117 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T07:26:10.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T07:26:10.511 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:10.728 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T07:26:10.800 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T07:26:10.906 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T07:26:10.995 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T07:26:11.156 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T07:26:11.363 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T07:26:11.520 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T07:26:11.716 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T07:26:11.876 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T07:26:12.046 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T07:26:12.255 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T07:26:12.450 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T07:26:12.525 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T07:26:12.645 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T07:26:12.745 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T07:26:12.887 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T07:26:13.177 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T07:26:13.246 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T07:26:13.445 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T07:26:13.492 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T07:26:13.667 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T07:26:13.672 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T07:26:13.855 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T07:26:13.872 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T07:26:13.930 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T07:26:14.013 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T07:26:14.099 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T07:26:14.406 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T07:26:14.441 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T07:26:14.442 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T07:26:14.443 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T07:26:14.443 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T07:26:14.443 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T07:26:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T07:26:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T07:26:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T07:26:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T07:26:14.603 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T07:26:14.800 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T07:26:15.005 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T07:26:15.221 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T07:26:15.316 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T07:26:15.316 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T07:26:15.317 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T07:26:15.318 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:15.319 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:15.447 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T07:26:15.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-01-21T07:26:15.657 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T07:26:15.995 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T07:26:16.286 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T07:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:11 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:16.675 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T07:26:16.879 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T07:26:17.793 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:26:17.840 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:17.841 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:26:17.841 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:17.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-01-21T07:26:18.414 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T07:26:18.587 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T07:26:18.656 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:13 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:19.019 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T07:26:19.262 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T07:26:19.483 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T07:26:19.639 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T07:26:19.657 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T07:26:19.805 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T07:26:20.117 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:26:20.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:20.119 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 71 k 2024-01-21T07:26:20.119 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 290 k 2024-01-21T07:26:20.119 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:20.167 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T07:26:20.167 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:20.168 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-21T07:26:20.168 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:20.175 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:20.175 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:20.176 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:20.179 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T07:26:20.212 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T07:26:20.213 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T07:26:20.214 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T07:26:20.250 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:20.747 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:20.922 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:21.020 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T07:26:21.020 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:21.020 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:21.020 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:21.020 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T07:26:21.021 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T07:26:21.022 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T07:26:21.023 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T07:26:21.023 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T07:26:21.023 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:21.023 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:21.144 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:21.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:21.500 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:21.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T07:26:21.977 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:09 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:22.438 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:17 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:23.421 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:26:23.467 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:23.468 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:26:23.468 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:23.557 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2024-01-21T07:26:23.933 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:23.933 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:23.933 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-01-21T07:26:23.933 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:23.933 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 163 k 2024-01-21T07:26:23.935 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 587 k 2024-01-21T07:26:23.935 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:23.982 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.3 MB/s | 163 kB 00:00 2024-01-21T07:26:23.983 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:23.983 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.3 MB/s | 163 kB 00:00 2024-01-21T07:26:23.983 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:23.994 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:23.994 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:24.044 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:24.165 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:24.276 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:12 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:24.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:24.790 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:25.182 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:25.183 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:25.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-01-21T07:26:25.759 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:25.759 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:26:25.760 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:25.761 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 71 k 2024-01-21T07:26:25.761 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 290 k 2024-01-21T07:26:25.761 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:25.817 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.2 MB/s | 71 kB 00:00 2024-01-21T07:26:25.818 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:25.818 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.2 MB/s | 71 kB 00:00 2024-01-21T07:26:25.818 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:25.825 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:25.825 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:25.827 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:25.830 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:25.870 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:26.096 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:26.343 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:26.498 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:26.720 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:27.078 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:27.296 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T07:26:27.543 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:26:27.588 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:27.590 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:26:27.590 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:27.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T07:26:28.026 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:28.413 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:29.521 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:29.521 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:26:29.522 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:29.523 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 163 k 2024-01-21T07:26:29.523 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 587 k 2024-01-21T07:26:29.523 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:29.572 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.2 MB/s | 163 kB 00:00 2024-01-21T07:26:29.572 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:29.573 INFO:teuthology.orchestra.run.smithi115.stdout:Total 3.2 MB/s | 163 kB 00:00 2024-01-21T07:26:29.573 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:29.584 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:29.584 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:29.630 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:29.634 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:29.750 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:29.909 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:29.910 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T07:26:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:29.912 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.0 M 2024-01-21T07:26:29.913 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 15 M 2024-01-21T07:26:29.913 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:29.936 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:30.311 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 448 kB/s | 24 kB 00:00 2024-01-21T07:26:30.336 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:30.336 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 113 kB/s | 8.9 kB 00:00 2024-01-21T07:26:30.395 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-01-21T07:26:30.545 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 161 kB/s | 34 kB 00:00 2024-01-21T07:26:30.612 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 904 kB/s | 196 kB 00:00 2024-01-21T07:26:30.637 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 130 kB/s | 43 kB 00:00 2024-01-21T07:26:30.639 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:30.639 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.1 MB/s | 3.0 MB 00:00 2024-01-21T07:26:30.649 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:30.658 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:30.658 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:30.832 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:30.835 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:31.122 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:31.350 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:26:31.350 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:31.350 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:31.351 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T07:26:31.351 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:31.357 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T07:26:31.593 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T07:26:31.802 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T07:26:31.931 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T07:26:32.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2024-01-21T07:26:32.936 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T07:26:32.966 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:20 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:33.232 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:26:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:26:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T07:26:33.501 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T07:26:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T07:26:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T07:26:33.502 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T07:26:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T07:26:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:33.878 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:33.879 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:33.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T07:26:34.409 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:26:34.455 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:34.456 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:26:34.456 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:34.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T07:26:34.720 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:29 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:35.272 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:36.200 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T07:26:36.202 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout:Install 8 Packages 2024-01-21T07:26:36.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2024-01-21T07:26:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 210 M 2024-01-21T07:26:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:36.735 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:36.736 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:36.736 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:36.737 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-01-21T07:26:36.738 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.0 M 2024-01-21T07:26:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 15 M 2024-01-21T07:26:36.738 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:36.751 INFO:teuthology.orchestra.run.smithi107.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2024-01-21T07:26:37.243 INFO:teuthology.orchestra.run.smithi107.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 7.2 MB/s | 4.7 MB 00:00 2024-01-21T07:26:37.310 INFO:teuthology.orchestra.run.smithi107.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.9 MB 00:00 2024-01-21T07:26:37.335 INFO:teuthology.orchestra.run.smithi107.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-01-21T07:26:37.444 INFO:teuthology.orchestra.run.smithi107.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.4 MB/s | 3.7 MB 00:00 2024-01-21T07:26:37.702 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 54 kB/s | 24 kB 00:00 2024-01-21T07:26:37.852 INFO:teuthology.orchestra.run.smithi107.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 422 kB/s | 172 kB 00:00 2024-01-21T07:26:37.859 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 15 kB/s | 8.9 kB 00:00 2024-01-21T07:26:37.903 INFO:teuthology.orchestra.run.smithi107.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2024-01-21T07:26:38.077 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 3.2 MB/s | 2.7 MB 00:00 2024-01-21T07:26:38.185 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 103 kB/s | 34 kB 00:00 2024-01-21T07:26:38.253 INFO:teuthology.orchestra.run.smithi107.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-01-21T07:26:38.254 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:38.254 INFO:teuthology.orchestra.run.smithi107.stdout:Total 18 MB/s | 37 MB 00:02 2024-01-21T07:26:38.260 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2024-01-21T07:26:38.311 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 70 kB/s | 43 kB 00:00 2024-01-21T07:26:38.313 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:38.313 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.9 MB/s | 3.0 MB 00:01 2024-01-21T07:26:38.325 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:38.336 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:38.336 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:38.483 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:38.497 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:38.500 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:38.501 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:38.501 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:38.762 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:38.823 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:38.964 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T07:26:39.168 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T07:26:39.377 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T07:26:39.444 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:39.498 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T07:26:39.582 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T07:26:39.726 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T07:26:40.473 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T07:26:40.582 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T07:26:40.732 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T07:26:41.015 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T07:26:41.186 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T07:26:41.390 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T07:26:41.409 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T07:26:41.410 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T07:26:41.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:41.410 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:41.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T07:26:42.003 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:42.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T07:26:42.256 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:30 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:42.864 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T07:26:43.202 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T07:26:43.423 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T07:26:43.743 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:43.744 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T07:26:43.745 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================ 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout:Install 8 Packages 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:43.746 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 37 M 2024-01-21T07:26:43.747 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 210 M 2024-01-21T07:26:43.747 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:44.158 INFO:teuthology.orchestra.run.smithi115.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 63 MB/s | 8.6 MB 00:00 2024-01-21T07:26:44.716 INFO:teuthology.orchestra.run.smithi115.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.0 MB/s | 4.9 MB 00:00 2024-01-21T07:26:44.783 INFO:teuthology.orchestra.run.smithi115.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.7 MB 00:00 2024-01-21T07:26:44.859 INFO:teuthology.orchestra.run.smithi115.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.3 MB/s | 3.7 MB 00:00 2024-01-21T07:26:44.884 INFO:teuthology.orchestra.run.smithi115.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2024-01-21T07:26:45.199 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T07:26:45.260 INFO:teuthology.orchestra.run.smithi115.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 460 kB/s | 172 kB 00:00 2024-01-21T07:26:45.393 INFO:teuthology.orchestra.run.smithi115.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-01-21T07:26:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 17 MB/s | 14 MB 00:00 2024-01-21T07:26:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:45.602 INFO:teuthology.orchestra.run.smithi115.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-21T07:26:45.831 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:45.849 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:45.849 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:46.162 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:46.165 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T07:26:46.781 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:46.792 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T07:26:46.949 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T07:26:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T07:26:47.103 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T07:26:47.104 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T07:26:47.224 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:47.535 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:47.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-01-21T07:26:47.822 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T07:26:48.040 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:48.360 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:48.658 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:49.298 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T07:26:49.845 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:49.846 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:49.846 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:26:49.846 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T07:26:49.847 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.9 M 2024-01-21T07:26:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2024-01-21T07:26:49.849 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:50.017 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T07:26:50.305 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 492 kB/s | 27 kB 00:00 2024-01-21T07:26:50.324 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T07:26:50.422 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 116 kB/s | 20 kB 00:00 2024-01-21T07:26:50.447 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 135 kB/s | 27 kB 00:00 2024-01-21T07:26:50.481 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 720 kB/s | 126 kB 00:00 2024-01-21T07:26:50.531 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2024-01-21T07:26:50.598 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T07:26:50.807 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2024-01-21T07:26:50.832 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 403 kB/s | 155 kB 00:00 2024-01-21T07:26:50.899 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 90 kB/s | 33 kB 00:00 2024-01-21T07:26:51.166 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 45 kB/s | 16 kB 00:00 2024-01-21T07:26:51.291 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 177 kB/s | 81 kB 00:00 2024-01-21T07:26:51.383 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-01-21T07:26:51.416 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2024-01-21T07:26:51.542 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 244 kB/s | 61 kB 00:00 2024-01-21T07:26:51.545 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:51.545 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-01-21T07:26:51.603 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:51.619 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:51.619 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:51.708 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:51.711 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:52.229 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:26:52.449 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T07:26:52.456 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T07:26:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T07:26:52.898 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T07:26:53.136 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T07:26:53.344 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T07:26:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T07:26:53.747 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T07:26:53.923 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T07:26:53.949 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T07:26:54.191 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T07:26:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T07:26:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T07:26:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T07:26:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T07:26:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T07:26:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T07:26:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T07:26:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T07:26:54.707 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T07:26:54.707 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:54.707 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:54.708 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:26:54.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-rook 2024-01-21T07:26:55.220 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T07:26:55.477 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T07:26:55.538 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:26:55.728 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T07:26:55.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T07:26:56.104 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T07:26:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T07:26:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T07:26:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T07:26:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T07:26:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T07:26:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:56.850 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:26:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T07:26:56.850 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:56.851 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:26:56.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T07:26:57.027 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:26:57.028 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:26:57.029 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T07:26:57.030 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:57.031 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:26:57.032 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:26:57.032 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2024-01-21T07:26:57.032 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:26:57.033 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 1.9 M 2024-01-21T07:26:57.033 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 24 M 2024-01-21T07:26:57.033 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:26:57.468 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 577 kB/s | 27 kB 00:00 2024-01-21T07:26:57.559 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 144 kB/s | 20 kB 00:00 2024-01-21T07:26:57.610 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 141 kB/s | 27 kB 00:00 2024-01-21T07:26:57.644 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 719 kB/s | 126 kB 00:00 2024-01-21T07:26:57.677 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 936 kB/s | 110 kB 00:00 2024-01-21T07:26:57.701 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:26:57.795 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 283 kB/s | 33 kB 00:00 2024-01-21T07:26:57.836 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 391 kB/s | 16 kB 00:00 2024-01-21T07:26:57.903 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2024-01-21T07:26:57.945 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 141 kB/s | 43 kB 00:00 2024-01-21T07:26:57.979 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 420 kB/s | 155 kB 00:00 2024-01-21T07:26:58.037 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): python3-kubernetes-11.0.0-6.el8.noarch 8.7 MB/s | 1.2 MB 00:00 2024-01-21T07:26:58.129 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 349 kB/s | 64 kB 00:00 2024-01-21T07:26:58.154 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 359 kB/s | 61 kB 00:00 2024-01-21T07:26:58.157 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:58.157 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-01-21T07:26:58.214 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:26:58.229 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:26:58.229 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:26:58.317 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:26:58.320 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:26:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:26:59.073 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T07:26:59.192 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:26:59.193 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:59.193 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-01-21T07:26:59.193 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:59.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-01-21T07:26:59.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:26:59.195 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.8 M 2024-01-21T07:26:59.195 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2024-01-21T07:26:59.195 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:26:59.292 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T07:26:59.342 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2024-01-21T07:26:59.465 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T07:26:59.493 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 141 kB/s | 21 kB 00:00 2024-01-21T07:26:59.593 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2024-01-21T07:26:59.676 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2024-01-21T07:26:59.719 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 389 kB/s | 49 kB 00:00 2024-01-21T07:26:59.785 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.7 MB/s | 4.8 MB 00:00 2024-01-21T07:26:59.786 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T07:26:59.786 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:26:59.786 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.8 MB/s | 5.8 MB 00:00 2024-01-21T07:26:59.836 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:26:59.846 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:26:59.846 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:26:59.962 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:26:59.965 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:26:59.990 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T07:27:00.233 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:00.252 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T07:27:00.518 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T07:27:00.537 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T07:27:00.794 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T07:27:00.813 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T07:27:01.755 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T07:27:01.823 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T07:27:02.094 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T07:27:02.114 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T07:27:02.323 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T07:27:02.327 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T07:27:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T07:27:02.535 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:27:02.687 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T07:27:02.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T07:27:02.984 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T07:27:02.985 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T07:27:02.985 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T07:27:03.225 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T07:27:03.225 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:03.225 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:03.226 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T07:27:03.226 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T07:27:03.226 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T07:27:03.226 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:03.226 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:03.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T07:27:03.447 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:03.448 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:03.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T07:27:04.080 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:59 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:04.288 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:05.577 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:05.577 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:05.577 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-01-21T07:27:05.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:05.579 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 968 k 2024-01-21T07:27:05.579 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 3.2 M 2024-01-21T07:27:05.579 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:05.732 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 11 MB/s | 885 kB 00:00 2024-01-21T07:27:05.754 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T07:27:05.755 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:05.756 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 5.8 M 2024-01-21T07:27:05.757 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 24 M 2024-01-21T07:27:05.757 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:05.962 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 270 kB/s | 83 kB 00:00 2024-01-21T07:27:05.963 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:05.963 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-01-21T07:27:05.973 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:05.993 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:05.993 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:06.079 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:06.082 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:06.270 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:06.429 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-01-21T07:27:06.542 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T07:27:06.638 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.1 MB/s | 538 kB 00:00 2024-01-21T07:27:06.718 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 73 kB/s | 21 kB 00:00 2024-01-21T07:27:06.726 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T07:27:06.821 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-01-21T07:27:06.888 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 287 kB/s | 49 kB 00:00 2024-01-21T07:27:07.187 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T07:27:07.187 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T07:27:07.614 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T07:27:07.614 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:07.614 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:07.615 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T07:27:07.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:07.615 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:07.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-01-21T07:27:08.264 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 219 kB/s | 355 kB 00:01 2024-01-21T07:27:08.265 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:08.266 INFO:teuthology.orchestra.run.smithi115.stdout:Total 2.3 MB/s | 5.8 MB 00:02 2024-01-21T07:27:08.319 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:08.328 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:08.328 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:08.436 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:08.439 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:08.560 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:03 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:08.728 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:08.976 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T07:27:09.262 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T07:27:10.048 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:10.049 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:27:10.050 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:10.050 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 131 k 2024-01-21T07:27:10.050 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 439 k 2024-01-21T07:27:10.050 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:10.086 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 131 kB 00:00 2024-01-21T07:27:10.086 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:10.087 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.4 MB/s | 131 kB 00:00 2024-01-21T07:27:10.087 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:10.092 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:10.092 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:10.141 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:10.144 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:10.177 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T07:27:10.266 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:10.541 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:10.614 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T07:27:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:10.839 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T07:27:11.010 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:27:11.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:11.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:11.206 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:11.206 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:11.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:11.207 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:11.296 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T07:27:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T07:27:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T07:27:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T07:27:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T07:27:11.297 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T07:27:11.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:11.686 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:11.809 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2024-01-21T07:27:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:07 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:12.532 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:00 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:13.588 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:13.634 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:13.635 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:13.635 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:13.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-01-21T07:27:14.020 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-01-21T07:27:14.021 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:14.022 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 968 k 2024-01-21T07:27:14.022 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 3.2 M 2024-01-21T07:27:14.022 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:14.282 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-01-21T07:27:14.472 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:09 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:14.524 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 271 kB/s | 83 kB 00:00 2024-01-21T07:27:14.524 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:14.525 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.9 MB/s | 968 kB 00:00 2024-01-21T07:27:14.534 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:14.555 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:14.555 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:14.641 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:14.645 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:14.826 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:15.117 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T07:27:15.309 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T07:27:15.724 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T07:27:15.724 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T07:27:15.952 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 24 k 2024-01-21T07:27:15.953 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 94 k 2024-01-21T07:27:15.954 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:16.029 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 320 kB/s | 24 kB 00:00 2024-01-21T07:27:16.030 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:16.030 INFO:teuthology.orchestra.run.smithi107.stdout:Total 314 kB/s | 24 kB 00:00 2024-01-21T07:27:16.030 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:16.034 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:16.034 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:16.036 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:16.039 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:16.090 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:16.126 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:16.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:16.349 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2024-01-21T07:27:16.539 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:16.878 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:17.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-01-21T07:27:17.081 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:05 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:17.724 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:12 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:18.544 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:27:18.545 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:18.546 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 131 k 2024-01-21T07:27:18.546 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 439 k 2024-01-21T07:27:18.546 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:18.596 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2024-01-21T07:27:18.596 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:18.597 INFO:teuthology.orchestra.run.smithi115.stdout:Total 2.5 MB/s | 131 kB 00:00 2024-01-21T07:27:18.597 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:18.652 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:18.654 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:18.768 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:19.024 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:19.161 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:19.207 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:19.208 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:19.208 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:19.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-01-21T07:27:19.326 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:19.700 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:19.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2024-01-21T07:27:20.029 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:15 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:20.648 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:08 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:21.476 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:21.521 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:21.522 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:21.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-01-21T07:27:22.086 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:22.132 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:22.133 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:22.133 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:22.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2024-01-21T07:27:22.358 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:17 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:22.944 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:10 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:23.800 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:23.846 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:23.847 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:23.847 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:23.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-01-21T07:27:24.418 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:24.419 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:24.419 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:24.419 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:24.419 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:24.419 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 24 k 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 94 k 2024-01-21T07:27:24.420 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:24.459 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 628 kB/s | 24 kB 00:00 2024-01-21T07:27:24.459 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:24.459 INFO:teuthology.orchestra.run.smithi115.stdout:Total 609 kB/s | 24 kB 00:00 2024-01-21T07:27:24.460 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:24.463 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:24.463 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:24.465 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:24.468 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:24.542 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:24.677 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:19 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:24.714 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:24.983 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:25.356 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:25.357 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:25.357 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:25.357 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:25.357 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:25.357 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:25.460 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2024-01-21T07:27:26.085 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:26.130 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:26.132 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:26.132 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:26.186 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:14 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:26.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-01-21T07:27:26.942 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:27.631 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:27.676 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:27.677 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:27.677 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:27.783 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2024-01-21T07:27:28.365 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:28.410 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:28.411 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:28.411 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:28.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-01-21T07:27:28.506 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:29.225 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:24 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:29.941 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:29.986 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:29.987 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:29.987 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:30.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2024-01-21T07:27:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:30.714 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:30.715 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-01-21T07:27:30.715 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:30.811 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:30.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-01-21T07:27:31.537 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:26 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:32.239 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:32.284 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:32.285 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:32.285 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:32.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:33.002 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 89 k 2024-01-21T07:27:33.003 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 298 k 2024-01-21T07:27:33.004 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:33.079 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.1 MB/s | 89 kB 00:00 2024-01-21T07:27:33.079 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:33.080 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.1 MB/s | 89 kB 00:00 2024-01-21T07:27:33.080 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:33.089 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:33.090 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:33.118 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:33.138 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:33.141 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:33.256 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:33.494 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:34.181 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:34.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-01-21T07:27:34.548 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:34.594 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:34.595 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:34.595 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:34.697 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2024-01-21T07:27:35.085 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:35.423 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:36.548 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:36.549 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:36.550 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:27:36.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:36.550 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.7 M 2024-01-21T07:27:36.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 18 M 2024-01-21T07:27:36.550 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:36.637 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2024-01-21T07:27:36.638 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:36.638 INFO:teuthology.orchestra.run.smithi107.stdout:Total 42 MB/s | 3.7 MB 00:00 2024-01-21T07:27:36.638 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:36.650 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:36.650 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:36.712 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:36.715 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:36.871 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:36.917 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:36.918 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:36.918 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:37.022 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2024-01-21T07:27:37.180 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:37.387 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:37.747 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:38.266 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:38.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-01-21T07:27:39.173 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T07:27:39.197 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:34 ago on Sun 21 Jan 2024 07:25:05 AM UTC. 2024-01-21T07:27:39.215 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:39.216 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-01-21T07:27:39.217 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:39.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2024-01-21T07:27:40.035 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:40.678 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T07:27:40.679 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:40.680 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-01-21T07:27:40.680 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-01-21T07:27:40.680 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-01-21T07:27:40.680 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:40.681 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 173 k 2024-01-21T07:27:40.681 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 649 k 2024-01-21T07:27:40.681 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-01-21T07:27:40.733 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 173 kB 00:00 2024-01-21T07:27:40.734 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:40.734 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.1 MB/s | 173 kB 00:00 2024-01-21T07:27:40.734 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-01-21T07:27:40.745 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-01-21T07:27:40.745 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-01-21T07:27:40.794 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-01-21T07:27:40.797 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-01-21T07:27:40.906 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-01-21T07:27:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:41.426 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:41.512 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:27:41.514 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 89 k 2024-01-21T07:27:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 298 k 2024-01-21T07:27:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:41.564 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-01-21T07:27:41.564 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:41.565 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-21T07:27:41.565 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:41.575 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:41.575 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:41.624 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:41.627 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:41.733 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:41.822 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-01-21T07:27:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:41.823 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-01-21T07:27:41.823 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-01-21T07:27:41.927 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:42.025 DEBUG:teuthology.parallel:result is None 2024-01-21T07:27:42.218 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:42.602 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:43.447 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2024-01-21T07:27:44.186 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:32 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:45.662 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:45.662 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:45.662 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:45.662 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:45.663 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.7 M 2024-01-21T07:27:45.664 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 18 M 2024-01-21T07:27:45.664 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:45.744 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-01-21T07:27:45.745 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:45.745 INFO:teuthology.orchestra.run.smithi115.stdout:Total 45 MB/s | 3.7 MB 00:00 2024-01-21T07:27:45.746 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:45.757 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:45.757 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:45.820 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:45.823 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:46.528 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:46.941 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:47.311 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:47.311 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:47.312 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:47.312 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:47.312 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:47.312 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:47.576 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2024-01-21T07:27:48.314 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:36 ago on Sun 21 Jan 2024 07:25:12 AM UTC. 2024-01-21T07:27:49.774 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-01-21T07:27:49.774 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:49.774 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-01-21T07:27:49.774 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:49.774 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 173 k 2024-01-21T07:27:49.775 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 649 k 2024-01-21T07:27:49.776 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-01-21T07:27:49.822 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-01-21T07:27:49.823 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-01-21T07:27:49.823 INFO:teuthology.orchestra.run.smithi115.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-01-21T07:27:49.823 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-01-21T07:27:49.834 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-01-21T07:27:49.834 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-01-21T07:27:49.884 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-01-21T07:27:49.887 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-01-21T07:27:50.007 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-01-21T07:27:50.192 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:50.483 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:50.894 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T07:27:50.894 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:50.894 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-01-21T07:27:50.894 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T07:27:50.895 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:27:50.895 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-01-21T07:27:51.129 DEBUG:teuthology.parallel:result is None 2024-01-21T07:27:51.129 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:27:51.129 INFO:teuthology.packaging:ref: None 2024-01-21T07:27:51.130 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:27:51.130 INFO:teuthology.packaging:branch: None 2024-01-21T07:27:51.130 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:27:51.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T07:27:51.269 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T07:27:51.296 INFO:teuthology.orchestra.run.smithi107.stdout:16.2.4-0.el8 2024-01-21T07:27:51.296 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T07:27:51.296 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T07:27:51.298 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T07:27:51.298 INFO:teuthology.packaging:ref: None 2024-01-21T07:27:51.298 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T07:27:51.298 INFO:teuthology.packaging:branch: None 2024-01-21T07:27:51.298 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T07:27:51.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T07:27:51.508 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T07:27:51.533 INFO:teuthology.orchestra.run.smithi115.stdout:16.2.4-0.el8 2024-01-21T07:27:51.533 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T07:27:51.533 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T07:27:51.535 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T07:27:51.535 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:27:51.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T07:27:51.568 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:27:51.568 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T07:27:51.603 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T07:27:51.603 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:27:51.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T07:27:51.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T07:27:51.708 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:27:51.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T07:27:51.737 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T07:27:51.805 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T07:27:51.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:27:51.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T07:27:51.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T07:27:51.903 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:27:51.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T07:27:51.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T07:27:51.997 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T07:27:51.997 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:27:51.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T07:27:52.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T07:27:52.091 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:27:52.092 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T07:27:52.118 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T07:27:52.185 INFO:teuthology.run_tasks:Running task print... 2024-01-21T07:27:52.196 INFO:teuthology.task.print:**** done install task... 2024-01-21T07:27:52.196 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T07:27:52.275 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T07:27:52.275 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T07:27:52.275 INFO:tasks.cephadm:Cluster fsid is 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:27:52.275 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T07:27:52.276 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T07:27:52.276 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi107': '172.21.15.107', 'mon.smithi115': '172.21.15.115'} 2024-01-21T07:27:52.276 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T07:27:52.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-01-21T07:27:52.303 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-01-21T07:27:52.330 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T07:27:52.330 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.541 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 07:27 /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.541 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.675 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 07:27 /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.675 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.694 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T07:27:52.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-01-21T07:27:52.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-01-21T07:27:52.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-01-21T07:27:52.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-01-21T07:27:52.853 INFO:tasks.cephadm:Writing seed config... 2024-01-21T07:27:52.854 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T07:27:52.854 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T07:27:52.854 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T07:27:52.854 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T07:27:52.855 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T07:27:52.856 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T07:27:52.857 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T07:27:52.857 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T07:27:52.857 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T07:27:52.857 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T07:27:52.857 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:27:52.857 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T07:27:52.883 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 962f5a9a-b82e-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T07:27:52.884 DEBUG:teuthology.orchestra.run.smithi107:mon.smithi107> sudo journalctl -f -n 0 -u ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service 2024-01-21T07:27:52.926 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T07:27:52.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.107 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T07:27:52.951 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:-- Logs begin at Sun 2024-01-21 07:18:30 UTC. -- 2024-01-21T07:27:53.210 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T07:27:54.038 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.6.1 2024-01-21T07:27:54.048 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying podman|docker is present... 2024-01-21T07:27:54.048 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T07:27:54.070 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.6.1 2024-01-21T07:27:54.080 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying lvm2 is present... 2024-01-21T07:27:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying time synchronization is in place... 2024-01-21T07:27:54.081 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T07:27:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T07:27:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active chrony.service 2024-01-21T07:27:54.095 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:27:54.095 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T07:27:54.101 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: enabled 2024-01-21T07:27:54.102 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T07:27:54.107 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: active 2024-01-21T07:27:54.108 INFO:teuthology.orchestra.run.smithi107.stderr:Unit chronyd.service is enabled and running 2024-01-21T07:27:54.108 INFO:teuthology.orchestra.run.smithi107.stderr:Repeating the final host check... 2024-01-21T07:27:54.108 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T07:27:54.128 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 4.6.1 2024-01-21T07:27:54.142 INFO:teuthology.orchestra.run.smithi107.stderr:podman|docker (/bin/podman) is present 2024-01-21T07:27:54.142 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl is present 2024-01-21T07:27:54.142 INFO:teuthology.orchestra.run.smithi107.stderr:lvcreate is present 2024-01-21T07:27:54.142 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T07:27:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T07:27:54.148 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active chrony.service 2024-01-21T07:27:54.153 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:27:54.154 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T07:27:54.160 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: enabled 2024-01-21T07:27:54.160 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T07:27:54.165 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: active 2024-01-21T07:27:54.166 INFO:teuthology.orchestra.run.smithi107.stderr:Unit chronyd.service is enabled and running 2024-01-21T07:27:54.166 INFO:teuthology.orchestra.run.smithi107.stderr:Host looks OK 2024-01-21T07:27:54.166 INFO:teuthology.orchestra.run.smithi107.stderr:Cluster fsid: 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:27:54.166 INFO:teuthology.orchestra.run.smithi107.stderr:Acquiring lock 140162507057640 on /run/cephadm/962f5a9a-b82e-11ee-95b0-87774f69a715.lock 2024-01-21T07:27:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:Lock 140162507057640 acquired on /run/cephadm/962f5a9a-b82e-11ee-95b0-87774f69a715.lock 2024-01-21T07:27:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying IP 172.21.15.107 port 3300 ... 2024-01-21T07:27:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying IP 172.21.15.107 port 6789 ... 2024-01-21T07:27:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:Base mon IP is 172.21.15.107, final addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-01-21T07:27:54.167 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /sbin/ip route ls 2024-01-21T07:27:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-01-21T07:27:54.169 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-01-21T07:27:54.170 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T07:27:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T07:27:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-21T07:27:54.173 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-01-21T07:27:54.173 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T07:27:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T07:27:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T07:27:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T07:27:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-21T07:27:54.175 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-01-21T07:27:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T07:27:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:Mon IP 172.21.15.107 is in CIDR network 172.21.0.0/20 2024-01-21T07:27:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T07:27:54.178 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T07:27:54.178 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T07:27:54.267 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T07:28:00.839 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Getting image source signatures 2024-01-21T07:28:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T07:28:00.857 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T07:28:12.830 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T07:28:12.921 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T07:28:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T07:28:13.279 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T07:28:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T07:28:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T07:28:14.146 INFO:teuthology.orchestra.run.smithi107.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T07:28:14.146 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T07:28:14.429 INFO:teuthology.orchestra.run.smithi107.stderr:stat: 167 167 2024-01-21T07:28:14.760 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial keys... 2024-01-21T07:28:14.760 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T07:28:15.048 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQCPx6xl0J0ZABAA1dWMSzHVn4CxwmXs1XC3KQ== 2024-01-21T07:28:15.366 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T07:28:15.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQCPx6xljRPDJBAAbWkOUi5kAhY2PVesDsxtxA== 2024-01-21T07:28:15.958 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T07:28:16.233 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-authtool: AQCQx6xlUX1BDBAAp5Eome6mk+3gREFWWmMjlQ== 2024-01-21T07:28:16.564 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial monmap... 2024-01-21T07:28:16.564 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpedf_o0js:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 --addv smithi107 [v2:172.21.15.107:3300,v1:172.21.15.107:6789] /tmp/monmap 2024-01-21T07:28:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T07:28:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T07:28:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:monmaptool for smithi107 [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T07:28:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: set fsid to 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T07:28:17.169 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:28:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:Creating mon... 2024-01-21T07:28:17.169 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/mon.smithi107:/var/lib/ceph/mon/ceph-smithi107:z -v /tmp/ceph-tmp7c3ojtgb:/tmp/keyring:z -v /tmp/ceph-tmpedf_o0js:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi107 --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T07:28:17.481 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.449+0000 7f49b3947700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.449+0000 7f49b3947700 1 imported monmap: 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T07:28:16.804927+0000 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: created 2024-01-21T07:28:16.804927+0000 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.smithi107 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.449+0000 7f49b3947700 0 /usr/bin/ceph-mon: set fsid to 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.482 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Compile date May 13 2021 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: DB SUMMARY 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi107/store.db dir, Total Num: 0, files: 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi107/store.db: 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.env: 0x55de5b71f1c0 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.fs: Posix File System 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.info_log: 0x55de5db20ec0 2024-01-21T07:28:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.statistics: (nil) 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.use_fsync: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.db_log_dir: 2024-01-21T07:28:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi107/store.db 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.write_buffer_manager: 0x55de5db27530 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T07:28:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.unordered_write: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.row_cache: None 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.wal_filter: None 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T07:28:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T07:28:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_open_files: -1 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Compression algorithms supported: 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kZSTD supported: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T07:28:17.488 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi107/store.db/MANIFEST-000001 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.merge_operator: 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_filter: None 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55de5da38120) 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T07:28:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache: 0x55de5da6ed10 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T07:28:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T07:28:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression: NoCompression 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.num_levels: 7 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T07:28:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T07:28:17.493 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T07:28:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T07:28:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.ttl: 2592000 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.450+0000 7f49b3947700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.451+0000 7f49b3947700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi107/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.451+0000 7f49b3947700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.451+0000 7f49b3947700 4 rocksdb: DB pointer 0x55de5db33800 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.452+0000 7f499c7e0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.452+0000 7f499c7e0700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T07:28:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T07:28:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.498 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T07:28:17.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.453+0000 7f49b3947700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.453+0000 7f49b3947700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T07:28:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph-mon: debug 2024-01-21T07:28:17.453+0000 7f49b3947700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi107 for mon.smithi107 2024-01-21T07:28:17.815 INFO:teuthology.orchestra.run.smithi107.stderr:create mon.smithi107 on 2024-01-21T07:28:17.815 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:17.820 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl disable ceph.target 2024-01-21T07:28:17.826 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T07:28:17.907 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl enable ceph.target 2024-01-21T07:28:17.914 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T07:28:17.990 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl start ceph.target 2024-01-21T07:28:17.997 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl enable ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target 2024-01-21T07:28:18.003 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target. 2024-01-21T07:28:18.003 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target. 2024-01-21T07:28:18.080 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target 2024-01-21T07:28:18.087 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl daemon-reload 2024-01-21T07:28:18.171 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:28:18.194 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl reset-failed ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:28:18.199 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to reset failed state of unit ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service: Unit ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service not loaded. 2024-01-21T07:28:18.200 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl enable ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:28:18.207 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target.wants/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service → /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@.service. 2024-01-21T07:28:18.290 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:28:18.577 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T07:28:18.585 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-01-21T07:28:18.586 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T07:28:18.593 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:28:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-01-21T07:28:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T07:28:18.594 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T07:28:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mon to start... 2024-01-21T07:28:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mon... 2024-01-21T07:28:18.595 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/mon.smithi107:/var/lib/ceph/mon/ceph-smithi107:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T07:28:18.611 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:18 smithi107 bash[38335]: 63a9bb369d3df82797598b11aee758df54c961c92990634a5718ac656f6641e5 2024-01-21T07:28:18.611 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:18 smithi107 systemd[1]: Started Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715. 2024-01-21T07:28:18.996 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.994+0000 7f216f59e700 1 Processor -- start 2024-01-21T07:28:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.994+0000 7f216f59e700 1 -- start start 2024-01-21T07:28:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.994+0000 7f216f59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:18.997 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.994+0000 7f216f59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21700fb2e0 con 0x7f21700fa9d0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54656/0 (socket says 172.21.15.107:54656) 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216e59c700 1 -- 172.21.15.107:0/3644701489 learned_addr learned my addr 172.21.15.107:0/3644701489 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216e59c700 1 -- 172.21.15.107:0/3644701489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21700fb420 con 0x7f21700fa9d0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216e59c700 1 --2- 172.21.15.107:0/3644701489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f216000b5f0 tx=0x7f2160006ee0).ready entity=mon.0 client_cookie=a731f7f0e654b75b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216d59a700 1 -- 172.21.15.107:0/3644701489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f216000d600 con 0x7f21700fa9d0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216d59a700 1 -- 172.21.15.107:0/3644701489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216000d760 con 0x7f21700fa9d0 2024-01-21T07:28:18.998 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.995+0000 7f216d59a700 1 -- 172.21.15.107:0/3644701489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f216000da30 con 0x7f21700fa9d0 2024-01-21T07:28:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 -- 172.21.15.107:0/3644701489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 msgr2=0x7f21700fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 --2- 172.21.15.107:0/3644701489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f216000b5f0 tx=0x7f2160006ee0).stop 2024-01-21T07:28:18.999 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 -- 172.21.15.107:0/3644701489 shutdown_connections 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 --2- 172.21.15.107:0/3644701489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f21700fadf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 -- 172.21.15.107:0/3644701489 >> 172.21.15.107:0/3644701489 conn(0x7f21700f6210 msgr2=0x7f21700f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 -- 172.21.15.107:0/3644701489 shutdown_connections 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.996+0000 7f216f59e700 1 -- 172.21.15.107:0/3644701489 wait complete. 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216f59e700 1 Processor -- start 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216f59e700 1 -- start start 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216f59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216f59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217010a580 con 0x7f21700fa9d0 2024-01-21T07:28:19.000 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54660/0 (socket says 172.21.15.107:54660) 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.997+0000 7f216e59c700 1 -- 172.21.15.107:0/1741345526 learned_addr learned my addr 172.21.15.107:0/1741345526 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f216e59c700 1 -- 172.21.15.107:0/1741345526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216000b200 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f216e59c700 1 --2- 172.21.15.107:0/1741345526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2160006c70 tx=0x7f2160016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2160011b40 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160011ca0 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f216001d430 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217010a780 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217010ab90 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f216000f070 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.998+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2160027460 con 0x7f21700fa9d0 2024-01-21T07:28:19.001 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:18.999+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2170045ad0 con 0x7f21700fa9d0 2024-01-21T07:28:19.002 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.001+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2160016070 con 0x7f21700fa9d0 2024-01-21T07:28:19.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.606315+0000 mon.smithi107 (mon.0) 0 : [INF] mkfs 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:19.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.587959+0000 mon.smithi107 (mon.0) 1 : cluster [INF] mon.smithi107 is new leader, mons smithi107 in quorum (ranks 0) 2024-01-21T07:28:19.068 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.067+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2170103b40 con 0x7f21700fa9d0 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f215effd700 1 -- 172.21.15.107:0/1741345526 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f2160035160 con 0x7f21700fa9d0 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: cluster: 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: id: 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: services: 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi107 (age 0.462661s) 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T07:28:19.071 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: data: 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: pgs: 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 msgr2=0x7f217010a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 --2- 172.21.15.107:0/1741345526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2160006c70 tx=0x7f2160016000).stop 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 shutdown_connections 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 --2- 172.21.15.107:0/1741345526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21700fa9d0 0x7f217010a090 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 >> 172.21.15.107:0/1741345526 conn(0x7f21700f6210 msgr2=0x7f217006ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.068+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 shutdown_connections 2024-01-21T07:28:19.072 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.069+0000 7f216f59e700 1 -- 172.21.15.107:0/1741345526 wait complete. 2024-01-21T07:28:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:mon is available 2024-01-21T07:28:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T07:28:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z -v /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/mon.smithi107:/var/lib/ceph/mon/ceph-smithi107:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi107/config 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.803+0000 7f578e70a700 1 Processor -- start 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.803+0000 7f578e70a700 1 -- start start 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.803+0000 7f578e70a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.803+0000 7f578e70a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57880fee10 con 0x7f57880703f0 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.803+0000 7f5787fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5787fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40488/0 (socket says 172.21.15.107:40488) 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5787fff700 1 -- 172.21.15.107:0/2180287991 learned_addr learned my addr 172.21.15.107:0/2180287991 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5787fff700 1 -- 172.21.15.107:0/2180287991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880fef50 con 0x7f57880703f0 2024-01-21T07:28:19.806 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5787fff700 1 --2- 172.21.15.107:0/2180287991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f577c00b5f0 tx=0x7f577c006ee0).ready entity=mon.0 client_cookie=8dfb7560b7c2ac93 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:19.807 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5786ffd700 1 -- 172.21.15.107:0/2180287991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f577c00d600 con 0x7f57880703f0 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5786ffd700 1 -- 172.21.15.107:0/2180287991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c00d760 con 0x7f57880703f0 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.804+0000 7f5786ffd700 1 -- 172.21.15.107:0/2180287991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f577c00da30 con 0x7f57880703f0 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 -- 172.21.15.107:0/2180287991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 msgr2=0x7f57880fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 --2- 172.21.15.107:0/2180287991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f577c00b5f0 tx=0x7f577c006ee0).stop 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 -- 172.21.15.107:0/2180287991 shutdown_connections 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 --2- 172.21.15.107:0/2180287991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f57880fe920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 -- 172.21.15.107:0/2180287991 >> 172.21.15.107:0/2180287991 conn(0x7f57880f63c0 msgr2=0x7f57880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 -- 172.21.15.107:0/2180287991 shutdown_connections 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 -- 172.21.15.107:0/2180287991 wait complete. 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.805+0000 7f578e70a700 1 Processor -- start 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f578e70a700 1 -- start start 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f578e70a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:19.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f578e70a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578810a770 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5787fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5787fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40500/0 (socket says 172.21.15.107:40500) 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5787fff700 1 -- 172.21.15.107:0/2875571222 learned_addr learned my addr 172.21.15.107:0/2875571222 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5787fff700 1 -- 172.21.15.107:0/2875571222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c00b200 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5787fff700 1 --2- 172.21.15.107:0/2875571222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f577c006c70 tx=0x7f577c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f577c011b40 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c011ca0 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578810a970 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.806+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f577c01d430 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.807+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578810ad80 con 0x7f57880703f0 2024-01-21T07:28:19.809 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.807+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f577c00f070 con 0x7f57880703f0 2024-01-21T07:28:19.810 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.807+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f577c027460 con 0x7f57880703f0 2024-01-21T07:28:19.810 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.808+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5788045ad0 con 0x7f57880703f0 2024-01-21T07:28:19.811 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.810+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f577c016070 con 0x7f57880703f0 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.605661+0000 mon.smithi107 (mon.0) 2 : cluster [INF] mon.smithi107 is new leader, mons smithi107 in quorum (ranks 0) 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.605886+0000 mon.smithi107 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi107=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.608433+0000 mon.smithi107 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.611175+0000 mon.smithi107 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: cluster 2024-01-21T07:28:18.611717+0000 mon.smithi107 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T07:28:19.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: audit 2024-01-21T07:28:19.068283+0000 mon.smithi107 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.107:0/1741345526' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T07:28:19.874 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.873+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f57880ff1a0 con 0x7f57880703f0 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.876+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f577c01d6e0 con 0x7f57880703f0 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.876+0000 7f5784ff9700 1 -- 172.21.15.107:0/2875571222 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f577c036020 con 0x7f57880703f0 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [global] 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: fsid = 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-01-21T07:28:19.880 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [mgr] 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: [osd] 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T07:28:19.881 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 msgr2=0x7f578810a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 --2- 172.21.15.107:0/2875571222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f577c006c70 tx=0x7f577c016000).stop 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 shutdown_connections 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 --2- 172.21.15.107:0/2875571222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880703f0 0x7f578810a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 >> 172.21.15.107:0/2875571222 conn(0x7f57880f63c0 msgr2=0x7f57880f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.877+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 shutdown_connections 2024-01-21T07:28:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:19.878+0000 7f578e70a700 1 -- 172.21.15.107:0/2875571222 wait complete. 2024-01-21T07:28:20.234 INFO:teuthology.orchestra.run.smithi107.stderr:Generating new minimal ceph.conf... 2024-01-21T07:28:20.234 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z -v /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/mon.smithi107:/var/lib/ceph/mon/ceph-smithi107:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi107/config 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.573+0000 7f208c1bf700 1 Processor -- start 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.573+0000 7f208c1bf700 1 -- start start 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.573+0000 7f208c1bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.573+0000 7f208c1bf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840fee10 con 0x7f20840703f0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2089f5b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2089f5b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40512/0 (socket says 172.21.15.107:40512) 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2089f5b700 1 -- 172.21.15.107:0/1282580824 learned_addr learned my addr 172.21.15.107:0/1282580824 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2089f5b700 1 -- 172.21.15.107:0/1282580824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840fef50 con 0x7f20840703f0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2089f5b700 1 --2- 172.21.15.107:0/1282580824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f208000b5f0 tx=0x7f2080006ee0).ready entity=mon.0 client_cookie=72a5c9727a039071 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2088f59700 1 -- 172.21.15.107:0/1282580824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f208000d600 con 0x7f20840703f0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2088f59700 1 -- 172.21.15.107:0/1282580824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f208000dbd0 con 0x7f20840703f0 2024-01-21T07:28:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.574+0000 7f2088f59700 1 -- 172.21.15.107:0/1282580824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2080011c00 con 0x7f20840703f0 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- 172.21.15.107:0/1282580824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 msgr2=0x7f20840fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 --2- 172.21.15.107:0/1282580824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f208000b5f0 tx=0x7f2080006ee0).stop 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- 172.21.15.107:0/1282580824 shutdown_connections 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 --2- 172.21.15.107:0/1282580824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- 172.21.15.107:0/1282580824 >> 172.21.15.107:0/1282580824 conn(0x7f20840f63c0 msgr2=0x7f20840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- 172.21.15.107:0/1282580824 shutdown_connections 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- 172.21.15.107:0/1282580824 wait complete. 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 Processor -- start 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.575+0000 7f208c1bf700 1 -- start start 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f208c1bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f208c1bf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840fa820 con 0x7f20840703f0 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f2089f5b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f2089f5b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40526/0 (socket says 172.21.15.107:40526) 2024-01-21T07:28:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f2089f5b700 1 -- 172.21.15.107:0/1010292178 learned_addr learned my addr 172.21.15.107:0/1010292178 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f2089f5b700 1 -- 172.21.15.107:0/1010292178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208000b200 con 0x7f20840703f0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f2089f5b700 1 --2- 172.21.15.107:0/1010292178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f208000b190 tx=0x7f2080006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f208001b410 con 0x7f20840703f0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f208001b9e0 con 0x7f20840703f0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20840fa9c0 con 0x7f20840703f0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f208001a880 con 0x7f20840703f0 2024-01-21T07:28:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20840fadd0 con 0x7f20840703f0 2024-01-21T07:28:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f208000f070 con 0x7f20840703f0 2024-01-21T07:28:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.576+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2080025b70 con 0x7f20840703f0 2024-01-21T07:28:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.578+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2084045ad0 con 0x7f20840703f0 2024-01-21T07:28:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.580+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2080016070 con 0x7f20840703f0 2024-01-21T07:28:20.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.645+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f20840008d0 con 0x7f20840703f0 2024-01-21T07:28:20.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.645+0000 7f20727fc700 1 -- 172.21.15.107:0/1010292178 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f2080006a70 con 0x7f20840703f0 2024-01-21T07:28:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 msgr2=0x7f20840fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 --2- 172.21.15.107:0/1010292178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f208000b190 tx=0x7f2080006ee0).stop 2024-01-21T07:28:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 shutdown_connections 2024-01-21T07:28:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 --2- 172.21.15.107:0/1010292178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20840703f0 0x7f20840fc3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 >> 172.21.15.107:0/1010292178 conn(0x7f20840f63c0 msgr2=0x7f20840f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 shutdown_connections 2024-01-21T07:28:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:20.646+0000 7f208c1bf700 1 -- 172.21.15.107:0/1010292178 wait complete. 2024-01-21T07:28:20.943 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: audit 2024-01-21T07:28:19.873845+0000 mon.smithi107 (mon.0) 8 : audit [INF] from='client.? 172.21.15.107:0/2875571222' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T07:28:20.944 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: audit 2024-01-21T07:28:19.877141+0000 mon.smithi107 (mon.0) 9 : audit [INF] from='client.? 172.21.15.107:0/2875571222' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T07:28:20.992 INFO:teuthology.orchestra.run.smithi107.stderr:Restarting the monitor... 2024-01-21T07:28:20.992 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl restart ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:28:21.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:20 smithi107 systemd[1]: Stopping Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715... 2024-01-21T07:28:21.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: debug 2024-01-21T07:28:21.064+0000 7fb5715a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi107 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T07:28:21.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38350]: debug 2024-01-21T07:28:21.064+0000 7fb5715a2700 -1 mon.smithi107@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T07:28:21.691 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 podman[38676]: ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107 2024-01-21T07:28:21.691 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 systemd[1]: ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service: Succeeded. 2024-01-21T07:28:21.691 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 systemd[1]: Stopped Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715. 2024-01-21T07:28:21.691 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 systemd[1]: Starting Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715... 2024-01-21T07:28:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T07:28:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 bash[38747]: 952bcca252be08e046a29124749f60c570abb15735bfa012bc40e35899ad6d34 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.747+0000 7fe00d459700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.747+0000 7fe00d459700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.747+0000 7fe00d459700 0 pidfile_write: ignore empty --pid-file 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 systemd[1]: Started Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715. 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 0 load: jerasure load: lrc load: isa 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Compile date May 13 2021 2024-01-21T07:28:22.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: DB SUMMARY 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi107/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi107/store.db: 000006.log size: 92938 ; 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T07:28:22.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.env: 0x560a5b00c1c0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.fs: Posix File System 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.info_log: 0x560a5ba7f960 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.statistics: (nil) 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.use_fsync: 0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T07:28:22.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.db_log_dir: 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi107/store.db 2024-01-21T07:28:22.069 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.write_buffer_manager: 0x560a5ba85fb0 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T07:28:22.071 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.unordered_write: 0 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.row_cache: None 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.wal_filter: None 2024-01-21T07:28:22.072 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T07:28:22.073 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.max_open_files: -1 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Compression algorithms supported: 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kZSTD supported: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T07:28:22.074 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.752+0000 7fe00d459700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi107/store.db/MANIFEST-000005 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.merge_operator: 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_filter: None 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T07:28:22.075 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560a5b996128) 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cache_index_and_filter_blocks: 1 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: pin_top_level_index_and_filter: 1 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: index_type: 0 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: data_block_index_type: 0 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: index_shortening: 1 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: hash_index_allow_collision: 1 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: checksum: 1 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: no_block_cache: 0 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_cache: 0x560a5b9ccf10 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_cache_name: BinnedLRUCache 2024-01-21T07:28:22.076 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_cache_options: 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: capacity : 536870912 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: num_shard_bits : 4 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: strict_capacity_limit : 0 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: high_pri_pool_ratio: 0.000 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_cache_compressed: (nil) 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: persistent_cache: (nil) 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_size: 4096 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_size_deviation: 10 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_restart_interval: 16 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: index_block_restart_interval: 1 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: metadata_block_size: 4096 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: partition_filters: 0 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: use_delta_encoding: 1 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T07:28:22.077 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: whole_key_filtering: 1 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: verify_compression: 0 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: read_amp_bytes_per_bit: 0 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: format_version: 2 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: enable_index_compression: 1 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: block_align: 0 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression: NoCompression 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.num_levels: 7 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T07:28:22.078 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T07:28:22.079 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T07:28:22.080 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T07:28:22.081 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.ttl: 2592000 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T07:28:22.082 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi107/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705822101754137, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.753+0000 7fe00d459700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.754+0000 7fe00d459700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.754+0000 7fe00d459700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705822101755003, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705822101, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.754+0000 7fe00d459700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.755+0000 7fe00d459700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705822101755430, "job": 1, "event": "recovery_finished"} 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.755+0000 7fe00d459700 4 rocksdb: DB pointer 0x560a5ba91800 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.756+0000 7fdff4271700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.756+0000 7fdff4271700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T07:28:22.083 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** DB Stats ** 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** Compaction Stats [default] ** 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.084 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** Compaction Stats [default] ** 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Cumulative compaction: 0.00 GB write, 29.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Interval compaction: 0.00 GB write, 29.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.085 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** Compaction Stats [default] ** 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** Compaction Stats [default] ** 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.9 0.00 0.00 1 0.001 0 0 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:22.086 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Cumulative compaction: 0.00 GB write, 29.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.756+0000 7fe00d459700 0 starting mon.smithi107 rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi107 fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 1 mon.smithi107@-1(???) e1 preinit fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).mds e1 new map 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).mds e1 print_map 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: e1 2024-01-21T07:28:22.087 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: legacy client fscid: -1 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: No filesystems configured 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 0 mon.smithi107@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:19.873845+0000 mon.smithi107 (mon.0) 8 : audit [INF] from='client.? 172.21.15.107:0/2875571222' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:19.877141+0000 mon.smithi107 (mon.0) 9 : audit [INF] from='client.? 172.21.15.107:0/2875571222' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.757+0000 7fe00d459700 1 mon.smithi107@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expand map: {default=false} 2024-01-21T07:28:22.088 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expand map: {default=info} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:28:21.758+0000 7fe00d459700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:21.773988+0000 mon.smithi107 (mon.0) 1 : cluster [INF] mon.smithi107 is new leader, mons smithi107 in quorum (ranks 0) 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:21.774064+0000 mon.smithi107 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi107=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:21.774114+0000 mon.smithi107 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:21.774145+0000 mon.smithi107 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T07:28:22.089 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:21.774575+0000 mon.smithi107 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T07:28:22.093 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.079+0000 7fcc5ffe2700 1 Processor -- start 2024-01-21T07:28:22.093 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5ffe2700 1 -- start start 2024-01-21T07:28:22.093 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5ffe2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:22.093 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5ffe2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580fb490 con 0x7fcc580fab80 2024-01-21T07:28:22.093 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5dd7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5dd7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40542/0 (socket says 172.21.15.107:40542) 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.080+0000 7fcc5dd7e700 1 -- 172.21.15.107:0/872146738 learned_addr learned my addr 172.21.15.107:0/872146738 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5dd7e700 1 -- 172.21.15.107:0/872146738 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc580fb5d0 con 0x7fcc580fab80 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5dd7e700 1 --2- 172.21.15.107:0/872146738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fcc5400b5f0 tx=0x7fcc54006ee0).ready entity=mon.0 client_cookie=b8bd683a5318db19 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5cd7c700 1 -- 172.21.15.107:0/872146738 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5400d600 con 0x7fcc580fab80 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5cd7c700 1 -- 172.21.15.107:0/872146738 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcc5400dbd0 con 0x7fcc580fab80 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5cd7c700 1 -- 172.21.15.107:0/872146738 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc54011c00 con 0x7fcc580fab80 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/872146738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 msgr2=0x7fcc580fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.081+0000 7fcc5ffe2700 1 --2- 172.21.15.107:0/872146738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fcc5400b5f0 tx=0x7fcc54006ee0).stop 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/872146738 shutdown_connections 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 --2- 172.21.15.107:0/872146738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580fab80 0x7fcc580fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/872146738 >> 172.21.15.107:0/872146738 conn(0x7fcc580f63c0 msgr2=0x7fcc580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/872146738 shutdown_connections 2024-01-21T07:28:22.094 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/872146738 wait complete. 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 Processor -- start 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.082+0000 7fcc5ffe2700 1 -- start start 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5ffe2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5ffe2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5810dce0 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5dd7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5dd7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40544/0 (socket says 172.21.15.107:40544) 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5dd7e700 1 -- 172.21.15.107:0/2484124201 learned_addr learned my addr 172.21.15.107:0/2484124201 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5dd7e700 1 -- 172.21.15.107:0/2484124201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5400b200 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5dd7e700 1 --2- 172.21.15.107:0/2484124201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcc54011820 tx=0x7fcc54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5401b410 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcc5401b9e0 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5810de20 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5401a920 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.083+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5810e1d0 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.084+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fcc5400f070 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.084+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcc54024780 con 0x7fcc58070400 2024-01-21T07:28:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.084+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc58045ad0 con 0x7fcc58070400 2024-01-21T07:28:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.086+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fcc54016070 con 0x7fcc58070400 2024-01-21T07:28:22.150 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.149+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fcc580fc9a0 con 0x7fcc58070400 2024-01-21T07:28:22.155 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.154+0000 7fcc467fc700 1 -- 172.21.15.107:0/2484124201 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcc5401aa80 con 0x7fcc58070400 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 msgr2=0x7fcc5810d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 --2- 172.21.15.107:0/2484124201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcc54011820 tx=0x7fcc54006ee0).stop 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 shutdown_connections 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 --2- 172.21.15.107:0/2484124201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc58070400 0x7fcc5810d7f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 >> 172.21.15.107:0/2484124201 conn(0x7fcc580f63c0 msgr2=0x7fcc580f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 shutdown_connections 2024-01-21T07:28:22.157 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:22.155+0000 7fcc5ffe2700 1 -- 172.21.15.107:0/2484124201 wait complete. 2024-01-21T07:28:22.533 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T07:28:22.534 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T07:28:22.534 INFO:teuthology.orchestra.run.smithi107.stderr:Creating mgr... 2024-01-21T07:28:22.534 INFO:teuthology.orchestra.run.smithi107.stderr:Verifying port 9283 ... 2024-01-21T07:28:22.536 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl daemon-reload 2024-01-21T07:28:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht 2024-01-21T07:28:22.658 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl reset-failed ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht 2024-01-21T07:28:22.663 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Failed to reset failed state of unit ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht.service: Unit ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht.service not loaded. 2024-01-21T07:28:22.664 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl enable ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht 2024-01-21T07:28:22.670 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: Created symlink /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715.target.wants/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht.service → /etc/systemd/system/ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@.service. 2024-01-21T07:28:22.753 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mgr.smithi107.adejht 2024-01-21T07:28:23.084 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T07:28:23.093 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-01-21T07:28:23.093 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T07:28:23.100 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:28:23.101 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-01-21T07:28:23.101 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T07:28:23.101 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T07:28:23.101 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T07:28:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-01-21T07:28:23.109 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T07:28:23.116 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:28:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-01-21T07:28:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T07:28:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr to start... 2024-01-21T07:28:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr... 2024-01-21T07:28:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53eb59e700 1 Processor -- start 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53eb59e700 1 -- start start 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53eb59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53eb59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0fec20 con 0x7f53ec070430 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40562/0 (socket says 172.21.15.107:40562) 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.473+0000 7f53ea59c700 1 -- 172.21.15.107:0/1888925671 learned_addr learned my addr 172.21.15.107:0/1888925671 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.474+0000 7f53ea59c700 1 -- 172.21.15.107:0/1888925671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0fed60 con 0x7f53ec070430 2024-01-21T07:28:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.474+0000 7f53ea59c700 1 --2- 172.21.15.107:0/1888925671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f53dc00b5f0 tx=0x7f53dc006ee0).ready entity=mon.0 client_cookie=b2121259b8a01fdd server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.474+0000 7f53e959a700 1 -- 172.21.15.107:0/1888925671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc00d600 con 0x7f53ec070430 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.474+0000 7f53e959a700 1 -- 172.21.15.107:0/1888925671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53dc00dbd0 con 0x7f53ec070430 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.474+0000 7f53e959a700 1 -- 172.21.15.107:0/1888925671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc011c00 con 0x7f53ec070430 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- 172.21.15.107:0/1888925671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 msgr2=0x7f53ec0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 --2- 172.21.15.107:0/1888925671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f53dc00b5f0 tx=0x7f53dc006ee0).stop 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- 172.21.15.107:0/1888925671 shutdown_connections 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 --2- 172.21.15.107:0/1888925671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec0fe730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- 172.21.15.107:0/1888925671 >> 172.21.15.107:0/1888925671 conn(0x7f53ec0f6210 msgr2=0x7f53ec0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- 172.21.15.107:0/1888925671 shutdown_connections 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- 172.21.15.107:0/1888925671 wait complete. 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 Processor -- start 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- start start 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.475+0000 7f53eb59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec10ad20 con 0x7f53ec070430 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40578/0 (socket says 172.21.15.107:40578) 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53ea59c700 1 -- 172.21.15.107:0/3661013800 learned_addr learned my addr 172.21.15.107:0/3661013800 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53ea59c700 1 -- 172.21.15.107:0/3661013800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc00b200 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53ea59c700 1 --2- 172.21.15.107:0/3661013800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f53dc00b190 tx=0x7f53dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc01b410 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53dc01b9e0 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ec10af20 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc01a880 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec10b330 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f53dc00f070 con 0x7f53ec070430 2024-01-21T07:28:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.476+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f53dc025c90 con 0x7f53ec070430 2024-01-21T07:28:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.477+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ec045ad0 con 0x7f53ec070430 2024-01-21T07:28:23.481 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.480+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f53dc016070 con 0x7f53ec070430 2024-01-21T07:28:23.556 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.549+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f53ec104640 con 0x7f53ec070430 2024-01-21T07:28:23.556 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.549+0000 7f53daffd700 1 -- 172.21.15.107:0/3661013800 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f53dc01bb50 con 0x7f53ec070430 2024-01-21T07:28:23.556 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "962f5a9a-b82e-11ee-95b0-87774f69a715", 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "smithi107" 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:23.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T07:28:23.559 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": false, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-01-21T07:28:18.606722+0000", 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 msgr2=0x7f53ec10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 --2- 172.21.15.107:0/3661013800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f53dc00b190 tx=0x7f53dc006ee0).stop 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 shutdown_connections 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 --2- 172.21.15.107:0/3661013800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ec070430 0x7f53ec10a830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 >> 172.21.15.107:0/3661013800 conn(0x7f53ec0f6210 msgr2=0x7f53ec0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 shutdown_connections 2024-01-21T07:28:23.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:23.550+0000 7f53eb59e700 1 -- 172.21.15.107:0/3661013800 wait complete. 2024-01-21T07:28:23.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:22.154902+0000 mon.smithi107 (mon.0) 6 : audit [INF] from='client.? 172.21.15.107:0/2484124201' entity='client.admin' 2024-01-21T07:28:23.957 INFO:teuthology.orchestra.run.smithi107.stderr:mgr not available, waiting (1/15)... 2024-01-21T07:28:24.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:23.549528+0000 mon.smithi107 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.107:0/3661013800' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T07:28:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T07:28:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f27259cc700 1 Processor -- start 2024-01-21T07:28:26.321 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f27259cc700 1 -- start start 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f27259cc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f27259cc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2720072920 con 0x7f2720072500 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f271effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f271effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40586/0 (socket says 172.21.15.107:40586) 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.319+0000 7f271effd700 1 -- 172.21.15.107:0/3297886772 learned_addr learned my addr 172.21.15.107:0/3297886772 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.320+0000 7f271effd700 1 -- 172.21.15.107:0/3297886772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720070f90 con 0x7f2720072500 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.320+0000 7f271effd700 1 --2- 172.21.15.107:0/3297886772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f271000b5f0 tx=0x7f2710006ee0).ready entity=mon.0 client_cookie=7215cbc12192660e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.320+0000 7f271dffb700 1 -- 172.21.15.107:0/3297886772 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f271000d600 con 0x7f2720072500 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.320+0000 7f271dffb700 1 -- 172.21.15.107:0/3297886772 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f271000dbd0 con 0x7f2720072500 2024-01-21T07:28:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.320+0000 7f271dffb700 1 -- 172.21.15.107:0/3297886772 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2710011c00 con 0x7f2720072500 2024-01-21T07:28:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 -- 172.21.15.107:0/3297886772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 msgr2=0x7f2720070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 --2- 172.21.15.107:0/3297886772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f271000b5f0 tx=0x7f2710006ee0).stop 2024-01-21T07:28:26.324 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 -- 172.21.15.107:0/3297886772 shutdown_connections 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 --2- 172.21.15.107:0/3297886772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f2720070aa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 -- 172.21.15.107:0/3297886772 >> 172.21.15.107:0/3297886772 conn(0x7f27200f74b0 msgr2=0x7f27200f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 -- 172.21.15.107:0/3297886772 shutdown_connections 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.321+0000 7f27259cc700 1 -- 172.21.15.107:0/3297886772 wait complete. 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.322+0000 7f27259cc700 1 Processor -- start 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.322+0000 7f27259cc700 1 -- start start 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.322+0000 7f27259cc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.322+0000 7f27259cc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272010d1f0 con 0x7f2720072500 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f271effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f271effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40598/0 (socket says 172.21.15.107:40598) 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f271effd700 1 -- 172.21.15.107:0/1878952653 learned_addr learned my addr 172.21.15.107:0/1878952653 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f271effd700 1 -- 172.21.15.107:0/1878952653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271000b200 con 0x7f2720072500 2024-01-21T07:28:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f271effd700 1 --2- 172.21.15.107:0/1878952653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2710011820 tx=0x7f2710006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f271001b410 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f271001b9e0 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27259cc700 1 -- 172.21.15.107:0/1878952653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272010d3f0 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f271001a8b0 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27259cc700 1 -- 172.21.15.107:0/1878952653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272010d800 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f271000f070 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.323+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2710025d00 con 0x7f2720072500 2024-01-21T07:28:26.326 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.324+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8004fa0 con 0x7f2720072500 2024-01-21T07:28:26.328 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.326+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2710016070 con 0x7f2720072500 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.395+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f26f8005670 con 0x7f2720072500 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.396+0000 7f27077fe700 1 -- 172.21.15.107:0/1878952653 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f2710025460 con 0x7f2720072500 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "962f5a9a-b82e-11ee-95b0-87774f69a715", 2024-01-21T07:28:26.403 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "smithi107" 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T07:28:26.404 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T07:28:26.405 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": false, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T07:28:26.406 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-01-21T07:28:18.606722+0000", 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 msgr2=0x7f272010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 --2- 172.21.15.107:0/1878952653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2710011820 tx=0x7f2710006ee0).stop 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 shutdown_connections 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 --2- 172.21.15.107:0/1878952653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2720072500 0x7f272010cd00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:26.407 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 >> 172.21.15.107:0/1878952653 conn(0x7f27200f74b0 msgr2=0x7f27200f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 shutdown_connections 2024-01-21T07:28:26.408 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:26.397+0000 7f27057fa700 1 -- 172.21.15.107:0/1878952653 wait complete. 2024-01-21T07:28:26.692 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:26 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:26.396104+0000 mon.smithi107 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.107:0/1878952653' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T07:28:26.757 INFO:teuthology.orchestra.run.smithi107.stderr:mgr not available, waiting (2/15)... 2024-01-21T07:28:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:27.969158+0000 mon.smithi107 (mon.0) 9 : cluster [INF] Activating manager daemon smithi107.adejht 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:27.971839+0000 mon.smithi107 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi107.adejht(active, starting, since 0.00278391s) 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.972963+0000 mon.smithi107 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.973139+0000 mon.smithi107 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.973310+0000 mon.smithi107 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.973475+0000 mon.smithi107 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata", "id": "smithi107"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.973600+0000 mon.smithi107 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mgr metadata", "who": "smithi107.adejht", "id": "smithi107.adejht"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:27.986689+0000 mon.smithi107 (mon.0) 16 : cluster [INF] Manager daemon smithi107.adejht is now available 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.995033+0000 mon.smithi107 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.998337+0000 mon.smithi107 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:27.999453+0000 mon.smithi107 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:28.004200+0000 mon.smithi107 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' 2024-01-21T07:28:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:28.006071+0000 mon.smithi107 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.107:0/2995394380' entity='mgr.smithi107.adejht' 2024-01-21T07:28:28.760 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.094+0000 7fabca416700 1 Processor -- start 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.094+0000 7fabca416700 1 -- start start 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.094+0000 7fabca416700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.094+0000 7fabca416700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40fc5c0 con 0x7fabc40fbcb0 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40642/0 (socket says 172.21.15.107:40642) 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc3fff700 1 -- 172.21.15.107:0/1854616959 learned_addr learned my addr 172.21.15.107:0/1854616959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc3fff700 1 -- 172.21.15.107:0/1854616959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40fc700 con 0x7fabc40fbcb0 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc3fff700 1 --2- 172.21.15.107:0/1854616959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fabb400b5f0 tx=0x7fabb4006ee0).ready entity=mon.0 client_cookie=35120f8e152ee2c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:29.100 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc2ffd700 1 -- 172.21.15.107:0/1854616959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb400d600 con 0x7fabc40fbcb0 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc2ffd700 1 -- 172.21.15.107:0/1854616959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabb400dbd0 con 0x7fabc40fbcb0 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.095+0000 7fabc2ffd700 1 -- 172.21.15.107:0/1854616959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb4011c00 con 0x7fabc40fbcb0 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- 172.21.15.107:0/1854616959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 msgr2=0x7fabc40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 --2- 172.21.15.107:0/1854616959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fabb400b5f0 tx=0x7fabb4006ee0).stop 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- 172.21.15.107:0/1854616959 shutdown_connections 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 --2- 172.21.15.107:0/1854616959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40fc0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- 172.21.15.107:0/1854616959 >> 172.21.15.107:0/1854616959 conn(0x7fabc40f74b0 msgr2=0x7fabc40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- 172.21.15.107:0/1854616959 shutdown_connections 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- 172.21.15.107:0/1854616959 wait complete. 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 Processor -- start 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.096+0000 7fabca416700 1 -- start start 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabca416700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabca416700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc4070a60 con 0x7fabc40fbcb0 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40648/0 (socket says 172.21.15.107:40648) 2024-01-21T07:28:29.101 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc3fff700 1 -- 172.21.15.107:0/210372681 learned_addr learned my addr 172.21.15.107:0/210372681 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc3fff700 1 -- 172.21.15.107:0/210372681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb400b200 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc3fff700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fabb400e040 tx=0x7fabb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb401b410 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc4070c00 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fabb401b9e0 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb401a820 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.097+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc4071010 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.098+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fabb400f070 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.098+0000 7fabc0ff9700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 0x7fabb0032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.098+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fabb4041c30 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.098+0000 7fabc37fe700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 0x7fabb0032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.098+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4059100 con 0x7fabc40fbcb0 2024-01-21T07:28:29.102 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.099+0000 7fabc37fe700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 0x7fabb0032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fabb80096a0 tx=0x7fabb8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:29.104 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.103+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabb4016070 con 0x7fabc40fbcb0 2024-01-21T07:28:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.304+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fabc4045ad0 con 0x7fabc40fbcb0 2024-01-21T07:28:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.305+0000 7fabc0ff9700 1 -- 172.21.15.107:0/210372681 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fabb4025460 con 0x7fabc40fbcb0 2024-01-21T07:28:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsid": "962f5a9a-b82e-11ee-95b0-87774f69a715", 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "health": { 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 0 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "smithi107" 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T07:28:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T07:28:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "iostat", 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "restful" 2024-01-21T07:28:29.315 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ], 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "modified": "2024-01-21T07:28:18.606722+0000", 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "services": {} 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: }, 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.306+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 msgr2=0x7fabb0032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.306+0000 7fabca416700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 0x7fabb0032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fabb80096a0 tx=0x7fabb8006b40).stop 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.306+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 msgr2=0x7fabc40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.306+0000 7fabca416700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fabb400e040 tx=0x7fabb4006ee0).stop 2024-01-21T07:28:29.316 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 shutdown_connections 2024-01-21T07:28:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7fabb002fc50 0x7fabb0032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 --2- 172.21.15.107:0/210372681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabc40fbcb0 0x7fabc40725e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 >> 172.21.15.107:0/210372681 conn(0x7fabc40f74b0 msgr2=0x7fabc40f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 shutdown_connections 2024-01-21T07:28:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:29.307+0000 7fabca416700 1 -- 172.21.15.107:0/210372681 wait complete. 2024-01-21T07:28:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:mgr is available 2024-01-21T07:28:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling cephadm module... 2024-01-21T07:28:29.657 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T07:28:30.006 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433999c700 1 Processor -- start 2024-01-21T07:28:30.006 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433999c700 1 -- start start 2024-01-21T07:28:30.006 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433999c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433999c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340fb2e0 con 0x7f43340fa9d0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433899a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433899a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41598/0 (socket says 172.21.15.107:41598) 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433899a700 1 -- 172.21.15.107:0/59909548 learned_addr learned my addr 172.21.15.107:0/59909548 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433899a700 1 -- 172.21.15.107:0/59909548 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340fb420 con 0x7f43340fa9d0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.002+0000 7f433899a700 1 --2- 172.21.15.107:0/59909548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f432800b5f0 tx=0x7f4328006ee0).ready entity=mon.0 client_cookie=1b19da06100a752f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f43337fe700 1 -- 172.21.15.107:0/59909548 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f432800d600 con 0x7f43340fa9d0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f43337fe700 1 -- 172.21.15.107:0/59909548 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f432800dbd0 con 0x7f43340fa9d0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f43337fe700 1 -- 172.21.15.107:0/59909548 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4328011c00 con 0x7f43340fa9d0 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 -- 172.21.15.107:0/59909548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 msgr2=0x7f43340fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 --2- 172.21.15.107:0/59909548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f432800b5f0 tx=0x7f4328006ee0).stop 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 -- 172.21.15.107:0/59909548 shutdown_connections 2024-01-21T07:28:30.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 --2- 172.21.15.107:0/59909548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f43340fadf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 -- 172.21.15.107:0/59909548 >> 172.21.15.107:0/59909548 conn(0x7f43340f6210 msgr2=0x7f43340f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 -- 172.21.15.107:0/59909548 shutdown_connections 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.003+0000 7f433999c700 1 -- 172.21.15.107:0/59909548 wait complete. 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433999c700 1 Processor -- start 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433999c700 1 -- start start 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433999c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433999c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433410cf70 con 0x7f43340fa9d0 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433899a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433899a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41612/0 (socket says 172.21.15.107:41612) 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.004+0000 7f433899a700 1 -- 172.21.15.107:0/1210093049 learned_addr learned my addr 172.21.15.107:0/1210093049 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f433899a700 1 -- 172.21.15.107:0/1210093049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432800b200 con 0x7f43340fa9d0 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f433899a700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f432800e040 tx=0x7f4328006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f432801b410 con 0x7f43340fa9d0 2024-01-21T07:28:30.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433410d170 con 0x7f43340fa9d0 2024-01-21T07:28:30.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f432801b9e0 con 0x7f43340fa9d0 2024-01-21T07:28:30.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f432801a820 con 0x7f43340fa9d0 2024-01-21T07:28:30.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.005+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433410d580 con 0x7f43340fa9d0 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f432800f070 con 0x7f43340fa9d0 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f43317fa700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 0x7f4318032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4328041ff0 con 0x7f43340fa9d0 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f4333fff700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 0x7f4318032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4334059100 con 0x7f43340fa9d0 2024-01-21T07:28:30.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.006+0000 7f4333fff700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 0x7f4318032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f43200096a0 tx=0x7f4320006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:30.011 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.010+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4328016070 con 0x7f43340fa9d0 2024-01-21T07:28:30.195 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.194+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f43340fc6e0 con 0x7f43340fa9d0 2024-01-21T07:28:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:28.974055+0000 mon.smithi107 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi107.adejht(active, since 1.005s) 2024-01-21T07:28:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:29.305251+0000 mon.smithi107 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.107:0/210372681' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T07:28:30.974 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.973+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f43280257e0 con 0x7f43340fa9d0 2024-01-21T07:28:30.975 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.974+0000 7f43317fa700 1 -- 172.21.15.107:0/1210093049 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4328022310 con 0x7f43340fa9d0 2024-01-21T07:28:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 msgr2=0x7f4318032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 0x7f4318032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f43200096a0 tx=0x7f4320006b40).stop 2024-01-21T07:28:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 msgr2=0x7f433410ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f432800e040 tx=0x7f4328006ee0).stop 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 shutdown_connections 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f431802fc50 0x7f4318032110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 --2- 172.21.15.107:0/1210093049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43340fa9d0 0x7f433410ca80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 >> 172.21.15.107:0/1210093049 conn(0x7f43340f6210 msgr2=0x7f4334070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 shutdown_connections 2024-01-21T07:28:30.979 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:30.976+0000 7f433999c700 1 -- 172.21.15.107:0/1210093049 wait complete. 2024-01-21T07:28:31.258 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:30.194671+0000 mon.smithi107 (mon.0) 24 : audit [INF] from='client.? 172.21.15.107:0/1210093049' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T07:28:31.286 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T07:28:31.645 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc59f3700 1 Processor -- start 2024-01-21T07:28:31.645 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc59f3700 1 -- start start 2024-01-21T07:28:31.645 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc59f3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:31.645 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc59f3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00fc5a0 con 0x7f0bc00703b0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc49f1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc49f1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41632/0 (socket says 172.21.15.107:41632) 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.642+0000 7f0bc49f1700 1 -- 172.21.15.107:0/2816398237 learned_addr learned my addr 172.21.15.107:0/2816398237 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc49f1700 1 -- 172.21.15.107:0/2816398237 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc01010a0 con 0x7f0bc00703b0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc49f1700 1 --2- 172.21.15.107:0/2816398237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0bb400b5f0 tx=0x7f0bb4006ee0).ready entity=mon.0 client_cookie=febf560c8c5b5e34 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bbf7fe700 1 -- 172.21.15.107:0/2816398237 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb400d600 con 0x7f0bc00703b0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bbf7fe700 1 -- 172.21.15.107:0/2816398237 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0bb400dbd0 con 0x7f0bc00703b0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bbf7fe700 1 -- 172.21.15.107:0/2816398237 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb4011c00 con 0x7f0bc00703b0 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc59f3700 1 -- 172.21.15.107:0/2816398237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 msgr2=0x7f0bc0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/2816398237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0bb400b5f0 tx=0x7f0bb4006ee0).stop 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc59f3700 1 -- 172.21.15.107:0/2816398237 shutdown_connections 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/2816398237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc0100bb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.643+0000 7f0bc59f3700 1 -- 172.21.15.107:0/2816398237 >> 172.21.15.107:0/2816398237 conn(0x7f0bc00f6210 msgr2=0x7f0bc00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.644+0000 7f0bc59f3700 1 -- 172.21.15.107:0/2816398237 shutdown_connections 2024-01-21T07:28:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.644+0000 7f0bc59f3700 1 -- 172.21.15.107:0/2816398237 wait complete. 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc59f3700 1 Processor -- start 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc59f3700 1 -- start start 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc59f3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc59f3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc010cf60 con 0x7f0bc00703b0 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc49f1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc49f1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41640/0 (socket says 172.21.15.107:41640) 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc49f1700 1 -- 172.21.15.107:0/66770819 learned_addr learned my addr 172.21.15.107:0/66770819 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc49f1700 1 -- 172.21.15.107:0/66770819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb400b200 con 0x7f0bc00703b0 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.645+0000 7f0bc49f1700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0bb400e040 tx=0x7f0bb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb400d9f0 con 0x7f0bc00703b0 2024-01-21T07:28:31.649 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0bb401a9f0 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc010d160 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb401b8c0 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc010d570 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.646+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0bb400f070 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.647+0000 7f0bbd7fa700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 0x7f0ba4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.647+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0bb4042040 con 0x7f0bc00703b0 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.647+0000 7f0bbffff700 1 -- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 msgr2=0x7f0ba4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.647+0000 7f0bbffff700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 0x7f0ba4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T07:28:31.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.647+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0059100 con 0x7f0bc00703b0 2024-01-21T07:28:31.652 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.651+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bb4016070 con 0x7f0bc00703b0 2024-01-21T07:28:31.847 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.845+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0bc0101c00 con 0x7f0bc00703b0 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.846+0000 7f0bbd7fa700 1 -- 172.21.15.107:0/66770819 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f0bb4022d20 con 0x7f0bc00703b0 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 4, 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "active_name": "smithi107.adejht", 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T07:28:31.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:28:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bbffff700 1 -- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 msgr2=0x7f0ba4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bbffff700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 0x7f0ba4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T07:28:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 msgr2=0x7f0ba4032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T07:28:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 0x7f0ba4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 msgr2=0x7f0bc010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0bb400e040 tx=0x7f0bb4006ee0).stop 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 shutdown_connections 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7f0ba402fd50 0x7f0ba4032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 --2- 172.21.15.107:0/66770819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bc00703b0 0x7f0bc010ca70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 >> 172.21.15.107:0/66770819 conn(0x7f0bc00f6210 msgr2=0x7f0bc00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 shutdown_connections 2024-01-21T07:28:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:31.848+0000 7f0bc59f3700 1 -- 172.21.15.107:0/66770819 wait complete. 2024-01-21T07:28:32.141 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:31 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:30.974595+0000 mon.smithi107 (mon.0) 25 : audit [INF] from='client.? 172.21.15.107:0/1210093049' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T07:28:32.141 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:31 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:30.974647+0000 mon.smithi107 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi107.adejht(active, since 3s) 2024-01-21T07:28:32.141 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:31 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:31.846306+0000 mon.smithi107 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.107:0/66770819' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T07:28:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for the mgr to restart... 2024-01-21T07:28:32.180 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr epoch 4... 2024-01-21T07:28:32.181 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T07:28:32.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.553+0000 7effcbd83700 1 Processor -- start 2024-01-21T07:28:32.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effcbd83700 1 -- start start 2024-01-21T07:28:32.557 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effcbd83700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effcbd83700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40fedd0 con 0x7effc4070430 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effc9b1f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effc9b1f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41656/0 (socket says 172.21.15.107:41656) 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effc9b1f700 1 -- 172.21.15.107:0/826407436 learned_addr learned my addr 172.21.15.107:0/826407436 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effc9b1f700 1 -- 172.21.15.107:0/826407436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40fef10 con 0x7effc4070430 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.554+0000 7effc9b1f700 1 --2- 172.21.15.107:0/826407436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7effc000b5f0 tx=0x7effc0006ee0).ready entity=mon.0 client_cookie=f9c901d2100d10c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effc8b1d700 1 -- 172.21.15.107:0/826407436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc000d600 con 0x7effc4070430 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effc8b1d700 1 -- 172.21.15.107:0/826407436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7effc000dbd0 con 0x7effc4070430 2024-01-21T07:28:32.558 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effc8b1d700 1 -- 172.21.15.107:0/826407436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc0011c00 con 0x7effc4070430 2024-01-21T07:28:32.560 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 -- 172.21.15.107:0/826407436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 msgr2=0x7effc40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 --2- 172.21.15.107:0/826407436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7effc000b5f0 tx=0x7effc0006ee0).stop 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 -- 172.21.15.107:0/826407436 shutdown_connections 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 --2- 172.21.15.107:0/826407436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc40fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 -- 172.21.15.107:0/826407436 >> 172.21.15.107:0/826407436 conn(0x7effc40f63c0 msgr2=0x7effc40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 -- 172.21.15.107:0/826407436 shutdown_connections 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.555+0000 7effcbd83700 1 -- 172.21.15.107:0/826407436 wait complete. 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effcbd83700 1 Processor -- start 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effcbd83700 1 -- start start 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effcbd83700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effcbd83700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc410aef0 con 0x7effc4070430 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effc9b1f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effc9b1f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41660/0 (socket says 172.21.15.107:41660) 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effc9b1f700 1 -- 172.21.15.107:0/354602574 learned_addr learned my addr 172.21.15.107:0/354602574 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:32.561 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effc9b1f700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc000b200 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.556+0000 7effc9b1f700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7effc000b190 tx=0x7effc0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.557+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc001b410 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.557+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7effc001b9e0 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.557+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc001a880 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.557+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc410b0f0 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.557+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc410b500 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7effc000f070 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effb27fc700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7effac0328d0 con 0x7effac02fd50 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effc931e700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7effc0041210 con 0x7effc4070430 2024-01-21T07:28:32.562 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.558+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T07:28:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.759+0000 7effc931e700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:32.761 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:32.759+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T07:28:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:33.160+0000 7effc931e700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:33.160+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T07:28:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:33.961+0000 7effc931e700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:33.963 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:33.961+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T07:28:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:35.562+0000 7effc931e700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/553526196 2024-01-21T07:28:35.565 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:35.562+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T07:28:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:35.926+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7effc001b570 con 0x7effc4070430 2024-01-21T07:28:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:35.927+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 msgr2=0x7effac032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T07:28:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:35.927+0000 7effb27fc700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:35.876018+0000 mon.smithi107 (mon.0) 28 : cluster [INF] Active manager daemon smithi107.adejht restarted 2024-01-21T07:28:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:35.876398+0000 mon.smithi107 (mon.0) 29 : cluster [INF] Activating manager daemon smithi107.adejht 2024-01-21T07:28:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:35.878062+0000 mon.smithi107 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T07:28:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.933+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7effc0042f60 con 0x7effc4070430 2024-01-21T07:28:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.933+0000 7effb27fc700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 0x7effac035c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.934+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7effac0328d0 con 0x7effac033820 2024-01-21T07:28:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.934+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 0x7effac035c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:36.937 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.935+0000 7effc931e700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 0x7effac035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7effb4006fd0 tx=0x7effb4006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:36.938 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.936+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7effac0328d0 con 0x7effac033820 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7effc4059100 con 0x7effac033820 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effb27fc700 1 -- 172.21.15.107:0/354602574 <== mgr.14114 v2:172.21.15.107:6800/1859130614 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7effc4059100 con 0x7effac033820 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T07:28:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:28:36.948 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 msgr2=0x7effac035c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:36.948 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effcbd83700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 0x7effac035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7effb4006fd0 tx=0x7effb4006cf0).stop 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 msgr2=0x7effc410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.944+0000 7effcbd83700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7effc000b190 tx=0x7effc0006ee0).stop 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 shutdown_connections 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7effac033820 0x7effac035c10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effc4070430 0x7effc410aa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 --2- 172.21.15.107:0/354602574 >> [v2:172.21.15.107:6800/553526196,v1:172.21.15.107:6801/553526196] conn(0x7effac02fd50 0x7effac032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 >> 172.21.15.107:0/354602574 conn(0x7effc40f63c0 msgr2=0x7effc40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 shutdown_connections 2024-01-21T07:28:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:36.945+0000 7effcbd83700 1 -- 172.21.15.107:0/354602574 wait complete. 2024-01-21T07:28:37.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:35.929162+0000 mon.smithi107 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi107.adejht(active, starting, since 0.0528392s) 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.930675+0000 mon.smithi107 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata", "id": "smithi107"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.931258+0000 mon.smithi107 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mgr metadata", "who": "smithi107.adejht", "id": "smithi107.adejht"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.931998+0000 mon.smithi107 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.932128+0000 mon.smithi107 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.932260+0000 mon.smithi107 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:35.945624+0000 mon.smithi107 (mon.0) 37 : cluster [INF] Manager daemon smithi107.adejht is now available 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.960964+0000 mon.smithi107 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.966302+0000 mon.smithi107 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:35.971814+0000 mon.smithi107 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.059436+0000 mon.smithi107 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.060060+0000 mon.smithi107 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.065064+0000 mon.smithi107 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:37.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.066454+0000 mon.smithi107 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:37.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.072943+0000 mon.smithi107 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:28:37.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.074153+0000 mon.smithi107 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:28:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:mgr epoch 4 is available 2024-01-21T07:28:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:Setting orchestrator backend to cephadm... 2024-01-21T07:28:37.781 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.124+0000 7f4a51393700 1 Processor -- start 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.124+0000 7f4a51393700 1 -- start start 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.124+0000 7f4a51393700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.124+0000 7f4a51393700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0fee50 con 0x7f4a4c0703b0 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a4affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.130 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a4affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41732/0 (socket says 172.21.15.107:41732) 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a4affd700 1 -- 172.21.15.107:0/735314823 learned_addr learned my addr 172.21.15.107:0/735314823 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a4affd700 1 -- 172.21.15.107:0/735314823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c0fef90 con 0x7f4a4c0703b0 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a4affd700 1 --2- 172.21.15.107:0/735314823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4a4000b5f0 tx=0x7f4a40006ee0).ready entity=mon.0 client_cookie=96c84de295c3d12e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a49ffb700 1 -- 172.21.15.107:0/735314823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4000d600 con 0x7f4a4c0703b0 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a49ffb700 1 -- 172.21.15.107:0/735314823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a4000dbd0 con 0x7f4a4c0703b0 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.125+0000 7f4a49ffb700 1 -- 172.21.15.107:0/735314823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a40011c00 con 0x7f4a4c0703b0 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 -- 172.21.15.107:0/735314823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 msgr2=0x7f4a4c0fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 --2- 172.21.15.107:0/735314823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4a4000b5f0 tx=0x7f4a40006ee0).stop 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 -- 172.21.15.107:0/735314823 shutdown_connections 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 --2- 172.21.15.107:0/735314823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fe960 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 -- 172.21.15.107:0/735314823 >> 172.21.15.107:0/735314823 conn(0x7f4a4c0f63c0 msgr2=0x7f4a4c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 -- 172.21.15.107:0/735314823 shutdown_connections 2024-01-21T07:28:38.131 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 -- 172.21.15.107:0/735314823 wait complete. 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.126+0000 7f4a51393700 1 Processor -- start 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a51393700 1 -- start start 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a51393700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a51393700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0fa860 con 0x7f4a4c0703b0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a4affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a4affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41738/0 (socket says 172.21.15.107:41738) 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a4affd700 1 -- 172.21.15.107:0/2655338158 learned_addr learned my addr 172.21.15.107:0/2655338158 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a4affd700 1 -- 172.21.15.107:0/2655338158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4000b200 con 0x7f4a4c0703b0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a4affd700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4a4000b190 tx=0x7f4a40006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4001b410 con 0x7f4a4c0703b0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c0faa00 con 0x7f4a4c0703b0 2024-01-21T07:28:38.132 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a4001b9e0 con 0x7f4a4c0703b0 2024-01-21T07:28:38.133 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a4001a8f0 con 0x7f4a4c0703b0 2024-01-21T07:28:38.133 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.127+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c0fae10 con 0x7f4a4c0703b0 2024-01-21T07:28:38.133 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.128+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4a4000f070 con 0x7f4a4c0703b0 2024-01-21T07:28:38.133 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.128+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c045ad0 con 0x7f4a4c0703b0 2024-01-21T07:28:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.128+0000 7f4a337fe700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 0x7f4a38032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.128+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4a40041370 con 0x7f4a4c0703b0 2024-01-21T07:28:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.128+0000 7f4a4a7fc700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 0x7f4a38032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.129+0000 7f4a4a7fc700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 0x7f4a38032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4a3c0096a0 tx=0x7f4a3c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.132+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a400224f0 con 0x7f4a4c0703b0 2024-01-21T07:28:38.312 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.310+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a4c0ffcc0 con 0x7f4a3802fd50 2024-01-21T07:28:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.936788+0000 mgr.smithi107.adejht (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T07:28:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:36.937367+0000 mon.smithi107 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi107.adejht(active, since 1.06103s) 2024-01-21T07:28:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:36.945688+0000 mgr.smithi107.adejht (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T07:28:38.317 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.315+0000 7f4a337fe700 1 -- 172.21.15.107:0/2655338158 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a4c0ffcc0 con 0x7f4a3802fd50 2024-01-21T07:28:38.319 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 msgr2=0x7f4a38032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:38.319 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 0x7f4a38032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4a3c0096a0 tx=0x7f4a3c006b40).stop 2024-01-21T07:28:38.319 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 msgr2=0x7f4a4c0fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:38.319 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4a4000b190 tx=0x7f4a40006ee0).stop 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 shutdown_connections 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f4a3802fd50 0x7f4a38032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 --2- 172.21.15.107:0/2655338158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c0703b0 0x7f4a4c0fc3e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 >> 172.21.15.107:0/2655338158 conn(0x7f4a4c0f63c0 msgr2=0x7f4a4c0f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 shutdown_connections 2024-01-21T07:28:38.320 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.317+0000 7f4a51393700 1 -- 172.21.15.107:0/2655338158 wait complete. 2024-01-21T07:28:38.672 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T07:28:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.986+0000 7f3cb6c94700 1 Processor -- start 2024-01-21T07:28:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.986+0000 7f3cb6c94700 1 -- start start 2024-01-21T07:28:38.991 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.986+0000 7f3cb6c94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.986+0000 7f3cb6c94700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb00fee10 con 0x7f3cb00703f0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3cb4a30700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3cb4a30700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41748/0 (socket says 172.21.15.107:41748) 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3cb4a30700 1 -- 172.21.15.107:0/1352794520 learned_addr learned my addr 172.21.15.107:0/1352794520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3cb4a30700 1 -- 172.21.15.107:0/1352794520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00fef50 con 0x7f3cb00703f0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3cb4a30700 1 --2- 172.21.15.107:0/1352794520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3ca400b5f0 tx=0x7f3ca4006ee0).ready entity=mon.0 client_cookie=3656127e4058cf1a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3caf7fe700 1 -- 172.21.15.107:0/1352794520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca400d600 con 0x7f3cb00703f0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3caf7fe700 1 -- 172.21.15.107:0/1352794520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3ca400dbd0 con 0x7f3cb00703f0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.987+0000 7f3caf7fe700 1 -- 172.21.15.107:0/1352794520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca4011c00 con 0x7f3cb00703f0 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- 172.21.15.107:0/1352794520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 msgr2=0x7f3cb00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/1352794520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3ca400b5f0 tx=0x7f3ca4006ee0).stop 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- 172.21.15.107:0/1352794520 shutdown_connections 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/1352794520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb00fe920 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:38.992 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- 172.21.15.107:0/1352794520 >> 172.21.15.107:0/1352794520 conn(0x7f3cb00f63c0 msgr2=0x7f3cb00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- 172.21.15.107:0/1352794520 shutdown_connections 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- 172.21.15.107:0/1352794520 wait complete. 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 Processor -- start 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.988+0000 7f3cb6c94700 1 -- start start 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb6c94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb6c94700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb010af40 con 0x7f3cb00703f0 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb4a30700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb4a30700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41762/0 (socket says 172.21.15.107:41762) 2024-01-21T07:28:38.993 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb4a30700 1 -- 172.21.15.107:0/3515569629 learned_addr learned my addr 172.21.15.107:0/3515569629 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb4a30700 1 -- 172.21.15.107:0/3515569629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca400b200 con 0x7f3cb00703f0 2024-01-21T07:28:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb4a30700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3ca400e040 tx=0x7f3ca4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca400d9f0 con 0x7f3cb00703f0 2024-01-21T07:28:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3ca401a9f0 con 0x7f3cb00703f0 2024-01-21T07:28:38.994 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb010b140 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca401b8c0 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.989+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb010b550 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.990+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3ca400f070 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.990+0000 7f3cad7fa700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 0x7f3c94032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.990+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3ca40412c0 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.990+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb0045ad0 con 0x7f3cb00703f0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.990+0000 7f3caffff700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 0x7f3c94032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:38.995 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.991+0000 7f3caffff700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 0x7f3c94032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3c9c0096a0 tx=0x7f3c9c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:38.996 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:38.994+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ca4016070 con 0x7f3cb00703f0 2024-01-21T07:28:39.148 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.145+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cb010bdc0 con 0x7f3c9402fdb0 2024-01-21T07:28:39.148 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.146+0000 7f3cad7fa700 1 -- 172.21.15.107:0/3515569629 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3cb010bdc0 con 0x7f3c9402fdb0 2024-01-21T07:28:39.148 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: value unchanged 2024-01-21T07:28:39.150 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 msgr2=0x7f3c94032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:39.150 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 0x7f3c94032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3c9c0096a0 tx=0x7f3c9c006b40).stop 2024-01-21T07:28:39.150 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 msgr2=0x7f3cb010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3ca400e040 tx=0x7f3ca4006ee0).stop 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 shutdown_connections 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3c9402fdb0 0x7f3c94032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 --2- 172.21.15.107:0/3515569629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cb00703f0 0x7f3cb010aa50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 >> 172.21.15.107:0/3515569629 conn(0x7f3cb00f63c0 msgr2=0x7f3cb00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 shutdown_connections 2024-01-21T07:28:39.151 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.147+0000 7f3cb6c94700 1 -- 172.21.15.107:0/3515569629 wait complete. 2024-01-21T07:28:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:37.943926+0000 mon.smithi107 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi107.adejht(active, since 2s) 2024-01-21T07:28:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:38.316144+0000 mon.smithi107 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:38.319393+0000 mon.smithi107 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:39.521 INFO:teuthology.orchestra.run.smithi107.stderr:Generating ssh key... 2024-01-21T07:28:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T07:28:39.889 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.884+0000 7f3bf637d700 1 Processor -- start 2024-01-21T07:28:39.889 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.884+0000 7f3bf637d700 1 -- start start 2024-01-21T07:28:39.889 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.884+0000 7f3bf637d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.884+0000 7f3bf637d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00fb490 con 0x7f3bf00fab80 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48476/0 (socket says 172.21.15.107:48476) 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beffff700 1 -- 172.21.15.107:0/2672334776 learned_addr learned my addr 172.21.15.107:0/2672334776 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beffff700 1 -- 172.21.15.107:0/2672334776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf00fb5d0 con 0x7f3bf00fab80 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beffff700 1 --2- 172.21.15.107:0/2672334776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3be400b5f0 tx=0x7f3be4006ee0).ready entity=mon.0 client_cookie=805dfc9b3370a505 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beeffd700 1 -- 172.21.15.107:0/2672334776 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be400d600 con 0x7f3bf00fab80 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beeffd700 1 -- 172.21.15.107:0/2672334776 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3be400dbd0 con 0x7f3bf00fab80 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.885+0000 7f3beeffd700 1 -- 172.21.15.107:0/2672334776 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be4011c00 con 0x7f3bf00fab80 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- 172.21.15.107:0/2672334776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 msgr2=0x7f3bf00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 --2- 172.21.15.107:0/2672334776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3be400b5f0 tx=0x7f3be4006ee0).stop 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- 172.21.15.107:0/2672334776 shutdown_connections 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 --2- 172.21.15.107:0/2672334776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf00fafa0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:39.890 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- 172.21.15.107:0/2672334776 >> 172.21.15.107:0/2672334776 conn(0x7f3bf00f63c0 msgr2=0x7f3bf00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- 172.21.15.107:0/2672334776 shutdown_connections 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- 172.21.15.107:0/2672334776 wait complete. 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 Processor -- start 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.886+0000 7f3bf637d700 1 -- start start 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3bf637d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3bf637d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf010d1b0 con 0x7f3bf00fab80 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3beffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3beffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48482/0 (socket says 172.21.15.107:48482) 2024-01-21T07:28:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3beffff700 1 -- 172.21.15.107:0/125583756 learned_addr learned my addr 172.21.15.107:0/125583756 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3beffff700 1 -- 172.21.15.107:0/125583756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be400b200 con 0x7f3bf00fab80 2024-01-21T07:28:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3beffff700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3be400e040 tx=0x7f3be4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be401b410 con 0x7f3bf00fab80 2024-01-21T07:28:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3be401b9e0 con 0x7f3bf00fab80 2024-01-21T07:28:39.892 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf010d3b0 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be401a820 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.887+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf010d7c0 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.888+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3be400f070 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.888+0000 7f3becff9700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 0x7f3bdc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.888+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf0045ad0 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.888+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3be40413f0 con 0x7f3bf00fab80 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.888+0000 7f3bef7fe700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 0x7f3bdc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:39.893 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.889+0000 7f3bef7fe700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 0x7f3bdc032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3be00096a0 tx=0x7f3be0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:39.894 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:39.893+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3be4016070 con 0x7f3bf00fab80 2024-01-21T07:28:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.043+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bf010e020 con 0x7f3bdc02fd50 2024-01-21T07:28:40.160 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.157+0000 7f3becff9700 1 -- 172.21.15.107:0/125583756 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3bf010e020 con 0x7f3bdc02fd50 2024-01-21T07:28:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 msgr2=0x7f3bdc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 0x7f3bdc032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3be00096a0 tx=0x7f3be0006b40).stop 2024-01-21T07:28:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 msgr2=0x7f3bf010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3be400e040 tx=0x7f3be4006ee0).stop 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 shutdown_connections 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f3bdc02fd50 0x7f3bdc032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 --2- 172.21.15.107:0/125583756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bf00fab80 0x7f3bf010ccc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 >> 172.21.15.107:0/125583756 conn(0x7f3bf00f63c0 msgr2=0x7f3bf0070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 shutdown_connections 2024-01-21T07:28:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.159+0000 7f3bf637d700 1 -- 172.21.15.107:0/125583756 wait complete. 2024-01-21T07:28:40.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:38.312338+0000 mgr.smithi107.adejht (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:40.504 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T07:28:40.812 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.809+0000 7fbd79603700 1 Processor -- start 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.809+0000 7fbd79603700 1 -- start start 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.809+0000 7fbd79603700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.809+0000 7fbd79603700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740fb310 con 0x7fbd740faa00 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48492/0 (socket says 172.21.15.107:48492) 2024-01-21T07:28:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd72ffd700 1 -- 172.21.15.107:0/389182291 learned_addr learned my addr 172.21.15.107:0/389182291 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd72ffd700 1 -- 172.21.15.107:0/389182291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd740fb450 con 0x7fbd740faa00 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd72ffd700 1 --2- 172.21.15.107:0/389182291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbd6800b5f0 tx=0x7fbd68006ee0).ready entity=mon.0 client_cookie=7db1179cf447f38e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd71ffb700 1 -- 172.21.15.107:0/389182291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6800d600 con 0x7fbd740faa00 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd71ffb700 1 -- 172.21.15.107:0/389182291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd6800dbd0 con 0x7fbd740faa00 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.810+0000 7fbd71ffb700 1 -- 172.21.15.107:0/389182291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd68011c00 con 0x7fbd740faa00 2024-01-21T07:28:40.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 -- 172.21.15.107:0/389182291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 msgr2=0x7fbd740fae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 --2- 172.21.15.107:0/389182291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbd6800b5f0 tx=0x7fbd68006ee0).stop 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 -- 172.21.15.107:0/389182291 shutdown_connections 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 --2- 172.21.15.107:0/389182291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd740fae20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 -- 172.21.15.107:0/389182291 >> 172.21.15.107:0/389182291 conn(0x7fbd740f6240 msgr2=0x7fbd740f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 -- 172.21.15.107:0/389182291 shutdown_connections 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.811+0000 7fbd79603700 1 -- 172.21.15.107:0/389182291 wait complete. 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd79603700 1 Processor -- start 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd79603700 1 -- start start 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd79603700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd79603700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74070450 con 0x7fbd740faa00 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48494/0 (socket says 172.21.15.107:48494) 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd72ffd700 1 -- 172.21.15.107:0/2643345774 learned_addr learned my addr 172.21.15.107:0/2643345774 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd72ffd700 1 -- 172.21.15.107:0/2643345774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6800b200 con 0x7fbd740faa00 2024-01-21T07:28:40.819 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd72ffd700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fbd6800b1d0 tx=0x7fbd68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6801b410 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.812+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd6801b9e0 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.813+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd6801a840 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.813+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd74070650 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.813+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7410dc00 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.813+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbd6800f070 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.813+0000 7fbd5b7fe700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 0x7fbd60032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.814+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbd68041240 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.814+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd74045ad0 con 0x7fbd740faa00 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.814+0000 7fbd727fc700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 0x7fbd60032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.814+0000 7fbd727fc700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 0x7fbd60032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fbd640096a0 tx=0x7fbd64006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.818+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd68016070 con 0x7fbd740faa00 2024-01-21T07:28:40.973 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.971+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd740fc780 con 0x7fbd6002fd50 2024-01-21T07:28:40.973 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.971+0000 7fbd5b7fe700 1 -- 172.21.15.107:0/2643345774 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbd740fc780 con 0x7fbd6002fd50 2024-01-21T07:28:40.974 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgOZD2X3LgS9GyqBgGlKKvA5x6UEkHQzBDs/G8WLLKOXtScyneh9bUbz8t5i9ByAG8+X1xsWFSqwfhTCdvEaq1ucHCVhN6D9fQVdQbtSY/CDpxy+hQIfL+gOfFKaeSAH8a59S1gqAX455MS7T0Ob8c+2ut9z7LXCynuKf68WqYbcSmyQ8ez8SR25/l49nxlngV/FNUONVj5N3V3jAMxGvV32/DNXfsMO5ldcH8ALEpdXK+1DCLLSBIE3KrdRMas7hIj1gebuzg7RjMoASDaU+ygqPd7wtLtdquHVDyIkGMOhWyWuMeG9Febk9LLdORwlhGGjY80aFFGySK+4t+PPx0mpF5yixwR2UeWG8vbh0oYNDZ4c7GsAylqdR5VIawkww4E+aRYIg+PgutJ6r9UX9Dd17BL35bitWeg7hUHwgLFuFIBZF72ITpY7c0QsFFfkORgzGZuPGCZnSnpZuds108iAk2W4X6h8zH1QeeRs7/5Cdg52HHCoEwwm8g7jvw2iM= ceph-962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 msgr2=0x7fbd60032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 0x7fbd60032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fbd640096a0 tx=0x7fbd64006b40).stop 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 msgr2=0x7fbd7410d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fbd6800b1d0 tx=0x7fbd68006ee0).stop 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 shutdown_connections 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fbd6002fd50 0x7fbd60032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 --2- 172.21.15.107:0/2643345774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd740faa00 0x7fbd7410d550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 >> 172.21.15.107:0/2643345774 conn(0x7fbd740f6240 msgr2=0x7fbd740f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 shutdown_connections 2024-01-21T07:28:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:40.973+0000 7fbd79603700 1 -- 172.21.15.107:0/2643345774 wait complete. 2024-01-21T07:28:41.203 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:39.147055+0000 mgr.smithi107.adejht (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:41.203 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:40.156392+0000 mon.smithi107 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:41.203 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:40.158122+0000 mon.smithi107 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:41.362 INFO:teuthology.orchestra.run.smithi107.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T07:28:41.362 INFO:teuthology.orchestra.run.smithi107.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T07:28:41.363 INFO:teuthology.orchestra.run.smithi107.stderr:Adding host smithi107... 2024-01-21T07:28:41.363 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi107 172.21.15.107 2024-01-21T07:28:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.715+0000 7f5e93096700 1 Processor -- start 2024-01-21T07:28:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.715+0000 7f5e93096700 1 -- start start 2024-01-21T07:28:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.715+0000 7f5e93096700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.715+0000 7f5e93096700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c0fee10 con 0x7f5e8c0703f0 2024-01-21T07:28:41.719 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e90e32700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e90e32700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48504/0 (socket says 172.21.15.107:48504) 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e90e32700 1 -- 172.21.15.107:0/1865393876 learned_addr learned my addr 172.21.15.107:0/1865393876 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e90e32700 1 -- 172.21.15.107:0/1865393876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0fef50 con 0x7f5e8c0703f0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e90e32700 1 --2- 172.21.15.107:0/1865393876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5e8000b5f0 tx=0x7f5e80006ee0).ready entity=mon.0 client_cookie=6a71c66cfd6ed7f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e8b7fe700 1 -- 172.21.15.107:0/1865393876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e8000d600 con 0x7f5e8c0703f0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e8b7fe700 1 -- 172.21.15.107:0/1865393876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e8000dbd0 con 0x7f5e8c0703f0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.716+0000 7f5e8b7fe700 1 -- 172.21.15.107:0/1865393876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e80011c00 con 0x7f5e8c0703f0 2024-01-21T07:28:41.720 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 -- 172.21.15.107:0/1865393876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 msgr2=0x7f5e8c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 --2- 172.21.15.107:0/1865393876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5e8000b5f0 tx=0x7f5e80006ee0).stop 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 -- 172.21.15.107:0/1865393876 shutdown_connections 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 --2- 172.21.15.107:0/1865393876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c0fe920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 -- 172.21.15.107:0/1865393876 >> 172.21.15.107:0/1865393876 conn(0x7f5e8c0f63c0 msgr2=0x7f5e8c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 -- 172.21.15.107:0/1865393876 shutdown_connections 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.717+0000 7f5e93096700 1 -- 172.21.15.107:0/1865393876 wait complete. 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.718+0000 7f5e93096700 1 Processor -- start 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.718+0000 7f5e93096700 1 -- start start 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.718+0000 7f5e93096700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.718+0000 7f5e93096700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c10aeb0 con 0x7f5e8c0703f0 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e90e32700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e90e32700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48510/0 (socket says 172.21.15.107:48510) 2024-01-21T07:28:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e90e32700 1 -- 172.21.15.107:0/2972038600 learned_addr learned my addr 172.21.15.107:0/2972038600 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e90e32700 1 -- 172.21.15.107:0/2972038600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8000b200 con 0x7f5e8c0703f0 2024-01-21T07:28:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e90e32700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5e8000b190 tx=0x7f5e80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e8001b410 con 0x7f5e8c0703f0 2024-01-21T07:28:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c10b0b0 con 0x7f5e8c0703f0 2024-01-21T07:28:41.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e8001b9e0 con 0x7f5e8c0703f0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e8001a880 con 0x7f5e8c0703f0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.719+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c10b4c0 con 0x7f5e8c0703f0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5e8000f070 con 0x7f5e8c0703f0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e897fa700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 0x7f5e70032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5e80041270 con 0x7f5e8c0703f0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e8bfff700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 0x7f5e70032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e8bfff700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 0x7f5e70032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5e780096a0 tx=0x7f5e78006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.720+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c059100 con 0x7f5e8c0703f0 2024-01-21T07:28:41.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.725+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e80010360 con 0x7f5e8c0703f0 2024-01-21T07:28:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:41.895+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e8c1048f0 con 0x7f5e7002fd50 2024-01-21T07:28:42.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:40.045742+0000 mgr.smithi107.adejht (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-21T07:28:42.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:40.972855+0000 mgr.smithi107.adejht (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Added host 'smithi107' 2024-01-21T07:28:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.504+0000 7f5e897fa700 1 -- 172.21.15.107:0/2972038600 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f5e8c1048f0 con 0x7f5e7002fd50 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 msgr2=0x7f5e70032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 0x7f5e70032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5e780096a0 tx=0x7f5e78006b40).stop 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 msgr2=0x7f5e8c10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5e8000b190 tx=0x7f5e80006ee0).stop 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 shutdown_connections 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f5e7002fd50 0x7f5e70032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 --2- 172.21.15.107:0/2972038600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e8c0703f0 0x7f5e8c10a9c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 >> 172.21.15.107:0/2972038600 conn(0x7f5e8c0f63c0 msgr2=0x7f5e8c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 shutdown_connections 2024-01-21T07:28:42.509 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:42.506+0000 7f5e93096700 1 -- 172.21.15.107:0/2972038600 wait complete. 2024-01-21T07:28:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:41.897382+0000 mgr.smithi107.adejht (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:43.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:42.504533+0000 mon.smithi107 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:43.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:42.505038+0000 mon.smithi107 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying mon service with default placement... 2024-01-21T07:28:44.295 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T07:28:44.298 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:42.504820+0000 mgr.smithi107.adejht (mgr.14114) 9 : cephadm [INF] Added host smithi107 2024-01-21T07:28:45.844 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.664+0000 7fe07dc3a700 1 Processor -- start 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.664+0000 7fe07dc3a700 1 -- start start 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.664+0000 7fe07dc3a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.664+0000 7fe07dc3a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780fb2e0 con 0x7fe0780fa9d0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe07cc38700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe07cc38700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48512/0 (socket says 172.21.15.107:48512) 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe07cc38700 1 -- 172.21.15.107:0/111790547 learned_addr learned my addr 172.21.15.107:0/111790547 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe07cc38700 1 -- 172.21.15.107:0/111790547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780fb420 con 0x7fe0780fa9d0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe07cc38700 1 --2- 172.21.15.107:0/111790547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe06c00b5f0 tx=0x7fe06c006ee0).ready entity=mon.0 client_cookie=54205c56609c5165 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe0777fe700 1 -- 172.21.15.107:0/111790547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c00d600 con 0x7fe0780fa9d0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe0777fe700 1 -- 172.21.15.107:0/111790547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe06c00dbd0 con 0x7fe0780fa9d0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.665+0000 7fe0777fe700 1 -- 172.21.15.107:0/111790547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c011c00 con 0x7fe0780fa9d0 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- 172.21.15.107:0/111790547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 msgr2=0x7fe0780fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:45.845 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/111790547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe06c00b5f0 tx=0x7fe06c006ee0).stop 2024-01-21T07:28:45.846 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- 172.21.15.107:0/111790547 shutdown_connections 2024-01-21T07:28:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/111790547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe0780fadf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:45.848 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- 172.21.15.107:0/111790547 >> 172.21.15.107:0/111790547 conn(0x7fe0780f6210 msgr2=0x7fe0780f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- 172.21.15.107:0/111790547 shutdown_connections 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- 172.21.15.107:0/111790547 wait complete. 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 Processor -- start 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.666+0000 7fe07dc3a700 1 -- start start 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07dc3a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07dc3a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078070470 con 0x7fe0780fa9d0 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07cc38700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07cc38700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48514/0 (socket says 172.21.15.107:48514) 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07cc38700 1 -- 172.21.15.107:0/2641725028 learned_addr learned my addr 172.21.15.107:0/2641725028 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07cc38700 1 -- 172.21.15.107:0/2641725028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c00b200 con 0x7fe0780fa9d0 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe07cc38700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe06c00e040 tx=0x7fe06c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:45.849 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.667+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c01b410 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.668+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe06c01b9e0 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.668+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe078070670 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.668+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c01a820 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.668+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07810dc00 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.668+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe078045ad0 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.669+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe06c00f070 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.669+0000 7fe0757fa700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 0x7fe05c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.669+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe06c041ac0 con 0x7fe0780fa9d0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.669+0000 7fe077fff700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 0x7fe05c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.669+0000 7fe077fff700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 0x7fe05c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:45.850 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.673+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe06c023d60 con 0x7fe0780fa9d0 2024-01-21T07:28:45.854 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.852+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fe078057f10 con 0x7fe05c02fd50 2024-01-21T07:28:45.858 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.856+0000 7fe0757fa700 1 -- 172.21.15.107:0/2641725028 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe078057f10 con 0x7fe05c02fd50 2024-01-21T07:28:45.858 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 msgr2=0x7fe05c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 0x7fe05c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).stop 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 msgr2=0x7fe07810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe06c00e040 tx=0x7fe06c006ee0).stop 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 shutdown_connections 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fe05c02fd50 0x7fe05c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 --2- 172.21.15.107:0/2641725028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0780fa9d0 0x7fe07810d550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 >> 172.21.15.107:0/2641725028 conn(0x7fe0780f6210 msgr2=0x7fe0780f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 shutdown_connections 2024-01-21T07:28:45.861 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:45.858+0000 7fe07dc3a700 1 -- 172.21.15.107:0/2641725028 wait complete. 2024-01-21T07:28:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:45.854560+0000 mgr.smithi107.adejht (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:47.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:45.855255+0000 mgr.smithi107.adejht (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T07:28:47.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:45.857404+0000 mon.smithi107 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:47.558 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying mgr service with default placement... 2024-01-21T07:28:47.558 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T07:28:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.972+0000 7f52a8980700 1 Processor -- start 2024-01-21T07:28:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.972+0000 7f52a8980700 1 -- start start 2024-01-21T07:28:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.972+0000 7f52a8980700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.972+0000 7f52a8980700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40fb490 con 0x7f52a40fab80 2024-01-21T07:28:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48528/0 (socket says 172.21.15.107:48528) 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a259c700 1 -- 172.21.15.107:0/802812468 learned_addr learned my addr 172.21.15.107:0/802812468 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a259c700 1 -- 172.21.15.107:0/802812468 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a40fb5d0 con 0x7f52a40fab80 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a259c700 1 --2- 172.21.15.107:0/802812468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5294009960 tx=0x7f52940092d0).ready entity=mon.0 client_cookie=466f398d469b189a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a159a700 1 -- 172.21.15.107:0/802812468 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529400b9c0 con 0x7f52a40fab80 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a159a700 1 -- 172.21.15.107:0/802812468 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5294011400 con 0x7f52a40fab80 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.973+0000 7f52a159a700 1 -- 172.21.15.107:0/802812468 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52940103a0 con 0x7f52a40fab80 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- 172.21.15.107:0/802812468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 msgr2=0x7f52a40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 --2- 172.21.15.107:0/802812468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5294009960 tx=0x7f52940092d0).stop 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- 172.21.15.107:0/802812468 shutdown_connections 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 --2- 172.21.15.107:0/802812468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a40fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- 172.21.15.107:0/802812468 >> 172.21.15.107:0/802812468 conn(0x7f52a40f63c0 msgr2=0x7f52a40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- 172.21.15.107:0/802812468 shutdown_connections 2024-01-21T07:28:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- 172.21.15.107:0/802812468 wait complete. 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 Processor -- start 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- start start 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.974+0000 7f52a8980700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a410aef0 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48544/0 (socket says 172.21.15.107:48544) 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a259c700 1 -- 172.21.15.107:0/4271938942 learned_addr learned my addr 172.21.15.107:0/4271938942 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a259c700 1 -- 172.21.15.107:0/4271938942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294009570 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a259c700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f529400ba90 tx=0x7f52940092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5294010870 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a410b0f0 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5294010e40 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529401ae10 con 0x7f52a40fab80 2024-01-21T07:28:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.975+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a410b500 con 0x7f52a40fab80 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.976+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f529402a070 con 0x7f52a40fab80 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.976+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a4045ad0 con 0x7f52a40fab80 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.976+0000 7f529affd700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 0x7f52840321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.976+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5294041b20 con 0x7f52a40fab80 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.976+0000 7f52a1d9b700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 0x7f52840321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.977+0000 7f52a1d9b700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 0x7f52840321c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f528c0096a0 tx=0x7f528c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:48.982+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5294018500 con 0x7f52a40fab80 2024-01-21T07:28:49.166 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.164+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f52a4104aa0 con 0x7f528402fd00 2024-01-21T07:28:49.169 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.167+0000 7f529affd700 1 -- 172.21.15.107:0/4271938942 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f52a4104aa0 con 0x7f528402fd00 2024-01-21T07:28:49.170 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T07:28:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 msgr2=0x7f52840321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 0x7f52840321c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f528c0096a0 tx=0x7f528c006b40).stop 2024-01-21T07:28:49.172 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 msgr2=0x7f52a410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f529400ba90 tx=0x7f52940092d0).stop 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 shutdown_connections 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f528402fd00 0x7f52840321c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 --2- 172.21.15.107:0/4271938942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52a40fab80 0x7f52a410aa00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 >> 172.21.15.107:0/4271938942 conn(0x7f52a40f63c0 msgr2=0x7f52a4070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 shutdown_connections 2024-01-21T07:28:49.173 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:49.169+0000 7f52a8980700 1 -- 172.21.15.107:0/4271938942 wait complete. 2024-01-21T07:28:50.018 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying crash service with default placement... 2024-01-21T07:28:50.019 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.452+0000 7f0710af7700 1 Processor -- start 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.452+0000 7f0710af7700 1 -- start start 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.452+0000 7f0710af7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.452+0000 7f0710af7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c0fc720 con 0x7f070c070460 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38546/0 (socket says 172.21.15.107:38546) 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070a59c700 1 -- 172.21.15.107:0/876256172 learned_addr learned my addr 172.21.15.107:0/876256172 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070a59c700 1 -- 172.21.15.107:0/876256172 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c101220 con 0x7f070c070460 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070a59c700 1 --2- 172.21.15.107:0/876256172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f06fc00b5f0 tx=0x7f06fc006ee0).ready entity=mon.0 client_cookie=77e8d017cbca322e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070959a700 1 -- 172.21.15.107:0/876256172 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc00d600 con 0x7f070c070460 2024-01-21T07:28:50.458 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070959a700 1 -- 172.21.15.107:0/876256172 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f06fc00dbd0 con 0x7f070c070460 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.453+0000 7f070959a700 1 -- 172.21.15.107:0/876256172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc011c00 con 0x7f070c070460 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- 172.21.15.107:0/876256172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 msgr2=0x7f070c100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 --2- 172.21.15.107:0/876256172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f06fc00b5f0 tx=0x7f06fc006ee0).stop 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- 172.21.15.107:0/876256172 shutdown_connections 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 --2- 172.21.15.107:0/876256172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c100d30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- 172.21.15.107:0/876256172 >> 172.21.15.107:0/876256172 conn(0x7f070c0f63b0 msgr2=0x7f070c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- 172.21.15.107:0/876256172 shutdown_connections 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- 172.21.15.107:0/876256172 wait complete. 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 Processor -- start 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- start start 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f0710af7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c10d0c0 con 0x7f070c070460 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.454+0000 7f070a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f070a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38560/0 (socket says 172.21.15.107:38560) 2024-01-21T07:28:50.459 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f070a59c700 1 -- 172.21.15.107:0/2212539575 learned_addr learned my addr 172.21.15.107:0/2212539575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f070a59c700 1 -- 172.21.15.107:0/2212539575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc00b200 con 0x7f070c070460 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f070a59c700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f06fc00b190 tx=0x7f06fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc01b410 con 0x7f070c070460 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f06fc01b9e0 con 0x7f070c070460 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c10d2c0 con 0x7f070c070460 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc01a880 con 0x7f070c070460 2024-01-21T07:28:50.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.455+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c10d6d0 con 0x7f070c070460 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.456+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f06fc00f070 con 0x7f070c070460 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.456+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070c045ad0 con 0x7f070c070460 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.456+0000 7f06faffd700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 0x7f06f0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.456+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f06fc041280 con 0x7f070c070460 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.456+0000 7f0709d9b700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 0x7f06f0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:50.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.457+0000 7f0709d9b700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 0x7f06f0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f07000096a0 tx=0x7f0700006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:50.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.461+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06fc019950 con 0x7f070c070460 2024-01-21T07:28:50.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:49.165898+0000 mgr.smithi107.adejht (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:50.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:49.166400+0000 mgr.smithi107.adejht (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T07:28:50.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:49.168615+0000 mon.smithi107 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.646+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f070c10e010 con 0x7f06f002fdb0 2024-01-21T07:28:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.650+0000 7f06faffd700 1 -- 172.21.15.107:0/2212539575 <== mgr.14114 v2:172.21.15.107:6800/1859130614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f070c10e010 con 0x7f06f002fdb0 2024-01-21T07:28:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 msgr2=0x7f06f0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 0x7f06f0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f07000096a0 tx=0x7f0700006b40).stop 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 msgr2=0x7f070c10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f06fc00b190 tx=0x7f06fc006ee0).stop 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 shutdown_connections 2024-01-21T07:28:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7f06f002fdb0 0x7f06f0032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 --2- 172.21.15.107:0/2212539575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f070c070460 0x7f070c10cbd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 >> 172.21.15.107:0/2212539575 conn(0x7f070c0f63b0 msgr2=0x7f070c0f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 shutdown_connections 2024-01-21T07:28:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:50.652+0000 7f0710af7700 1 -- 172.21.15.107:0/2212539575 wait complete. 2024-01-21T07:28:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling mgr prometheus module... 2024-01-21T07:28:51.111 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T07:28:51.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.462+0000 7fcf3e7e7700 1 Processor -- start 2024-01-21T07:28:51.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.462+0000 7fcf3e7e7700 1 -- start start 2024-01-21T07:28:51.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.462+0000 7fcf3e7e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:51.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.462+0000 7fcf3e7e7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380fedd0 con 0x7fcf38070430 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf37fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf37fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38570/0 (socket says 172.21.15.107:38570) 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf37fff700 1 -- 172.21.15.107:0/3054730306 learned_addr learned my addr 172.21.15.107:0/3054730306 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf37fff700 1 -- 172.21.15.107:0/3054730306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf380fef10 con 0x7fcf38070430 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf37fff700 1 --2- 172.21.15.107:0/3054730306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fcf2800b5f0 tx=0x7fcf28006ee0).ready entity=mon.0 client_cookie=8e0714102d5259da server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf36ffd700 1 -- 172.21.15.107:0/3054730306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf2800d600 con 0x7fcf38070430 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf36ffd700 1 -- 172.21.15.107:0/3054730306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf2800dc10 con 0x7fcf38070430 2024-01-21T07:28:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.463+0000 7fcf36ffd700 1 -- 172.21.15.107:0/3054730306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28011cd0 con 0x7fcf38070430 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3054730306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 msgr2=0x7fcf380fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3054730306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fcf2800b5f0 tx=0x7fcf28006ee0).stop 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3054730306 shutdown_connections 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3054730306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3054730306 >> 172.21.15.107:0/3054730306 conn(0x7fcf380f63c0 msgr2=0x7fcf380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3054730306 shutdown_connections 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3054730306 wait complete. 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.464+0000 7fcf3e7e7700 1 Processor -- start 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf3e7e7700 1 -- start start 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf3e7e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf3e7e7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380fa7e0 con 0x7fcf38070430 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf37fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf37fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38572/0 (socket says 172.21.15.107:38572) 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf37fff700 1 -- 172.21.15.107:0/3949604603 learned_addr learned my addr 172.21.15.107:0/3949604603 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf37fff700 1 -- 172.21.15.107:0/3949604603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2800b200 con 0x7fcf38070430 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf37fff700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcf28006800 tx=0x7fcf28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf28011df0 con 0x7fcf38070430 2024-01-21T07:28:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf380fa980 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf2801b8c0 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf2801a830 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.465+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf380fad90 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.466+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcf2800f070 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.466+0000 7fcf34ff9700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 0x7fcf24032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.466+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcf28040e90 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.466+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf38045ad0 con 0x7fcf38070430 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.467+0000 7fcf377fe700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 0x7fcf24032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:51.472 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.467+0000 7fcf377fe700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 0x7fcf24032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fcf2c0096a0 tx=0x7fcf2c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:51.473 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.471+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf28024410 con 0x7fcf38070430 2024-01-21T07:28:51.677 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:51.675+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fcf38104860 con 0x7fcf38070430 2024-01-21T07:28:52.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:50.648281+0000 mgr.smithi107.adejht (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:28:52.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:50.648949+0000 mgr.smithi107.adejht (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T07:28:52.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:50.651877+0000 mon.smithi107 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:52.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:51.091154+0000 mon.smithi107 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:52.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:51.315861+0000 mon.smithi107 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.654+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fcf28024790 con 0x7fcf38070430 2024-01-21T07:28:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.655+0000 7fcf34ff9700 1 -- 172.21.15.107:0/3949604603 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7fcf28022310 con 0x7fcf38070430 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 msgr2=0x7fcf24032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 0x7fcf24032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fcf2c0096a0 tx=0x7fcf2c006b40).stop 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 msgr2=0x7fcf380fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcf28006800 tx=0x7fcf28006ee0).stop 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 shutdown_connections 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fcf2402fd50 0x7fcf24032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 --2- 172.21.15.107:0/3949604603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf38070430 0x7fcf380fc360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 >> 172.21.15.107:0/3949604603 conn(0x7fcf380f63c0 msgr2=0x7fcf380f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 shutdown_connections 2024-01-21T07:28:52.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:52.657+0000 7fcf3e7e7700 1 -- 172.21.15.107:0/3949604603 wait complete. 2024-01-21T07:28:52.892 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:51.676996+0000 mon.smithi107 (mon.0) 60 : audit [INF] from='client.? 172.21.15.107:0/3949604603' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T07:28:52.892 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:51.980336+0000 mon.smithi107 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.107:0/4020153640' entity='mgr.smithi107.adejht' 2024-01-21T07:28:53.050 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying prometheus service with default placement... 2024-01-21T07:28:53.050 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd15e5bd700 1 Processor -- start 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd15e5bd700 1 -- start start 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd15e5bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd15e5bd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580fedd0 con 0x7fd158070430 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd157fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd157fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38584/0 (socket says 172.21.15.107:38584) 2024-01-21T07:28:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd157fff700 1 -- 172.21.15.107:0/3850358220 learned_addr learned my addr 172.21.15.107:0/3850358220 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.457+0000 7fd157fff700 1 -- 172.21.15.107:0/3850358220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580fef10 con 0x7fd158070430 2024-01-21T07:28:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd157fff700 1 --2- 172.21.15.107:0/3850358220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd14800b5f0 tx=0x7fd148006ee0).ready entity=mon.0 client_cookie=713c231a015a2fe9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd156ffd700 1 -- 172.21.15.107:0/3850358220 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd14800d600 con 0x7fd158070430 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd156ffd700 1 -- 172.21.15.107:0/3850358220 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd14800dc10 con 0x7fd158070430 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd156ffd700 1 -- 172.21.15.107:0/3850358220 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd148011cd0 con 0x7fd158070430 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 -- 172.21.15.107:0/3850358220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 msgr2=0x7fd1580fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/3850358220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd14800b5f0 tx=0x7fd148006ee0).stop 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 -- 172.21.15.107:0/3850358220 shutdown_connections 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/3850358220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fe8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 -- 172.21.15.107:0/3850358220 >> 172.21.15.107:0/3850358220 conn(0x7fd1580f63c0 msgr2=0x7fd1580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 -- 172.21.15.107:0/3850358220 shutdown_connections 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.458+0000 7fd15e5bd700 1 -- 172.21.15.107:0/3850358220 wait complete. 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd15e5bd700 1 Processor -- start 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd15e5bd700 1 -- start start 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd15e5bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd15e5bd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580fa7e0 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd157fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd157fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38598/0 (socket says 172.21.15.107:38598) 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.459+0000 7fd157fff700 1 -- 172.21.15.107:0/1741437700 learned_addr learned my addr 172.21.15.107:0/1741437700 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd157fff700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14800b200 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd157fff700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd148006800 tx=0x7fd148006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd148011df0 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd14801b8c0 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd14801a830 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1580fa980 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.460+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1580fad90 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd14800f070 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd154ff9700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd148040ee0 con 0x7fd158070430 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd1577fe700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1859130614 2024-01-21T07:28:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T07:28:53.466 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.461+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd158059100 con 0x7fd158070430 2024-01-21T07:28:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.466+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd148016070 con 0x7fd158070430 2024-01-21T07:28:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.648+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd158104930 con 0x7fd14402fe00 2024-01-21T07:28:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.662+0000 7fd1577fe700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1859130614 2024-01-21T07:28:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:53.662+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T07:28:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:54.063+0000 7fd1577fe700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1859130614 2024-01-21T07:28:54.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:52.656371+0000 mon.smithi107 (mon.0) 62 : audit [INF] from='client.? 172.21.15.107:0/3949604603' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T07:28:54.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:52.656423+0000 mon.smithi107 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi107.adejht(active, since 16s) 2024-01-21T07:28:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:54.063+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T07:28:54.866 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:54.864+0000 7fd1577fe700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1859130614 2024-01-21T07:28:54.866 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:54.864+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T07:28:56.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:56.465+0000 7fd1577fe700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/1859130614 2024-01-21T07:28:56.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:56.465+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T07:28:57.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:57.770+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd148024790 con 0x7fd158070430 2024-01-21T07:28:57.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:57.770+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 msgr2=0x7fd1440322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T07:28:57.773 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:57.770+0000 7fd154ff9700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:57.930 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:57.560169+0000 mon.smithi107 (mon.0) 64 : cluster [INF] Active manager daemon smithi107.adejht restarted 2024-01-21T07:28:57.931 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:57.560611+0000 mon.smithi107 (mon.0) 65 : cluster [INF] Activating manager daemon smithi107.adejht 2024-01-21T07:28:57.931 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:57.562257+0000 mon.smithi107 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T07:28:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.771+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd148043d60 con 0x7fd158070430 2024-01-21T07:28:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.771+0000 7fd154ff9700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 0x7fd144036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.771+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 --> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd158104930 con 0x7fd144033c20 2024-01-21T07:28:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.771+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 0x7fd144036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.772+0000 7fd1577fe700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 0x7fd144036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd14c0036d0 tx=0x7fd14c006ce0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:58.781 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.779+0000 7fd154ff9700 1 -- 172.21.15.107:0/1741437700 <== mgr.14142 v2:172.21.15.107:6800/2854824672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd158104930 con 0x7fd144033c20 2024-01-21T07:28:58.781 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T07:28:58.783 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.780+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 msgr2=0x7fd144036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:58.783 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.780+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 0x7fd144036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd14c0036d0 tx=0x7fd14c006ce0).stop 2024-01-21T07:28:58.783 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.780+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 msgr2=0x7fd1580fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:58.783 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.780+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd148006800 tx=0x7fd148006ee0).stop 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 shutdown_connections 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fd144033c20 0x7fd144036010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd158070430 0x7fd1580fc360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 --2- 172.21.15.107:0/1741437700 >> [v2:172.21.15.107:6800/1859130614,v1:172.21.15.107:6801/1859130614] conn(0x7fd14402fe00 0x7fd1440322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 >> 172.21.15.107:0/1741437700 conn(0x7fd1580f63c0 msgr2=0x7fd1580f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 shutdown_connections 2024-01-21T07:28:58.784 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:58.781+0000 7fd15e5bd700 1 -- 172.21.15.107:0/1741437700 wait complete. 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:57.772110+0000 mon.smithi107 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi107.adejht(active, starting, since 0.211583s) 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.772852+0000 mon.smithi107 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata", "id": "smithi107"}]: dispatch 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.773453+0000 mon.smithi107 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mgr metadata", "who": "smithi107.adejht", "id": "smithi107.adejht"}]: dispatch 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.774111+0000 mon.smithi107 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.774337+0000 mon.smithi107 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.774548+0000 mon.smithi107 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:57.787740+0000 mon.smithi107 (mon.0) 73 : cluster [INF] Manager daemon smithi107.adejht is now available 2024-01-21T07:28:58.981 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.884374+0000 mon.smithi107 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:58.982 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.885740+0000 mon.smithi107 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:28:58.982 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.899749+0000 mon.smithi107 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:28:58.982 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:57.902429+0000 mon.smithi107 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:28:59.225 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying grafana service with default placement... 2024-01-21T07:28:59.225 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T07:28:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.718+0000 7fa6caa68700 1 Processor -- start 2024-01-21T07:28:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.718+0000 7fa6caa68700 1 -- start start 2024-01-21T07:28:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.718+0000 7fa6caa68700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:59.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.718+0000 7fa6caa68700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40fedd0 con 0x7fa6c4070430 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c8804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c8804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45022/0 (socket says 172.21.15.107:45022) 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c8804700 1 -- 172.21.15.107:0/2105108752 learned_addr learned my addr 172.21.15.107:0/2105108752 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c8804700 1 -- 172.21.15.107:0/2105108752 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40fef10 con 0x7fa6c4070430 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c8804700 1 --2- 172.21.15.107:0/2105108752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa6b800b5f0 tx=0x7fa6b8006ee0).ready entity=mon.0 client_cookie=73616cefb4ea0bf9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c37fe700 1 -- 172.21.15.107:0/2105108752 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b800d600 con 0x7fa6c4070430 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c37fe700 1 -- 172.21.15.107:0/2105108752 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6b800dc10 con 0x7fa6c4070430 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.719+0000 7fa6c37fe700 1 -- 172.21.15.107:0/2105108752 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b8011cd0 con 0x7fa6c4070430 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 -- 172.21.15.107:0/2105108752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 msgr2=0x7fa6c40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2105108752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa6b800b5f0 tx=0x7fa6b8006ee0).stop 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 -- 172.21.15.107:0/2105108752 shutdown_connections 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2105108752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c40fe8e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 -- 172.21.15.107:0/2105108752 >> 172.21.15.107:0/2105108752 conn(0x7fa6c40f63c0 msgr2=0x7fa6c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 -- 172.21.15.107:0/2105108752 shutdown_connections 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 -- 172.21.15.107:0/2105108752 wait complete. 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.720+0000 7fa6caa68700 1 Processor -- start 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6caa68700 1 -- start start 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6caa68700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6caa68700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c410afd0 con 0x7fa6c4070430 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c8804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c8804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45026/0 (socket says 172.21.15.107:45026) 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c8804700 1 -- 172.21.15.107:0/2742148436 learned_addr learned my addr 172.21.15.107:0/2742148436 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c8804700 1 -- 172.21.15.107:0/2742148436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b800b200 con 0x7fa6c4070430 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c8804700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa6b800df90 tx=0x7fa6b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b8011da0 con 0x7fa6c4070430 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6b801b8c0 con 0x7fa6c4070430 2024-01-21T07:28:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.721+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c410b1d0 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.722+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b801a930 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.722+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c410b5e0 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.722+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c4045ad0 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.722+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fa6b800f070 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.723+0000 7fa6c17fa700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 0x7fa6a80321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.723+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa6b8040e80 con 0x7fa6c4070430 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.723+0000 7fa6c3fff700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 0x7fa6a80321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:28:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.723+0000 7fa6c3fff700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 0x7fa6a80321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa6b00096a0 tx=0x7fa6b0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:28:59.729 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.727+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6b8019d40 con 0x7fa6c4070430 2024-01-21T07:28:59.812 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.809+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa6b801baf0 con 0x7fa6c4070430 2024-01-21T07:28:59.914 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.912+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 --> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6c410bec0 con 0x7fa6a802fd10 2024-01-21T07:28:59.918 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.916+0000 7fa6c17fa700 1 -- 172.21.15.107:0/2742148436 <== mgr.14142 v2:172.21.15.107:6800/2854824672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa6c410bec0 con 0x7fa6a802fd10 2024-01-21T07:28:59.918 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T07:28:59.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.917+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 msgr2=0x7fa6a80321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:59.920 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.917+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 0x7fa6a80321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa6b00096a0 tx=0x7fa6b0006b40).stop 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.917+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 msgr2=0x7fa6c410aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.917+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa6b800df90 tx=0x7fa6b8006ee0).stop 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 shutdown_connections 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fa6a802fd10 0x7fa6a80321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 --2- 172.21.15.107:0/2742148436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6c4070430 0x7fa6c410aae0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 >> 172.21.15.107:0/2742148436 conn(0x7fa6c40f63c0 msgr2=0x7fa6c40f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 shutdown_connections 2024-01-21T07:28:59.921 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:28:59.918+0000 7fa6caa68700 1 -- 172.21.15.107:0/2742148436 wait complete. 2024-01-21T07:29:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:58.773928+0000 mon.smithi107 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi107.adejht(active, since 1.21341s) 2024-01-21T07:29:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:58.775717+0000 mgr.smithi107.adejht (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T07:29:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:58.779790+0000 mon.smithi107 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' 2024-01-21T07:29:00.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:28:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:58.810221+0000 mon.smithi107 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' 2024-01-21T07:29:00.634 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying node-exporter service with default placement... 2024-01-21T07:29:00.634 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T07:29:01.007 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f523667a700 1 Processor -- start 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f523667a700 1 -- start start 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f523667a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f523667a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52300fc710 con 0x7f52300703f0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f522ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f522ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45042/0 (socket says 172.21.15.107:45042) 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.001+0000 7f522ffff700 1 -- 172.21.15.107:0/207173499 learned_addr learned my addr 172.21.15.107:0/207173499 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f522ffff700 1 -- 172.21.15.107:0/207173499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230101210 con 0x7f52300703f0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f522ffff700 1 --2- 172.21.15.107:0/207173499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f522400b5f0 tx=0x7f5224006ee0).ready entity=mon.0 client_cookie=9aba0cd8ee5d2bfb server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f522effd700 1 -- 172.21.15.107:0/207173499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522400d600 con 0x7f52300703f0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f522effd700 1 -- 172.21.15.107:0/207173499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f522400dc10 con 0x7f52300703f0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f522effd700 1 -- 172.21.15.107:0/207173499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5224011cd0 con 0x7f52300703f0 2024-01-21T07:29:01.008 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f523667a700 1 -- 172.21.15.107:0/207173499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 msgr2=0x7f5230100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.002+0000 7f523667a700 1 --2- 172.21.15.107:0/207173499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f522400b5f0 tx=0x7f5224006ee0).stop 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 -- 172.21.15.107:0/207173499 shutdown_connections 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 --2- 172.21.15.107:0/207173499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f5230100d20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 -- 172.21.15.107:0/207173499 >> 172.21.15.107:0/207173499 conn(0x7f52300f63c0 msgr2=0x7f52300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 -- 172.21.15.107:0/207173499 shutdown_connections 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 -- 172.21.15.107:0/207173499 wait complete. 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 Processor -- start 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 -- start start 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.003+0000 7f523667a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f523667a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523010d220 con 0x7f52300703f0 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45050/0 (socket says 172.21.15.107:45050) 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522ffff700 1 -- 172.21.15.107:0/1372882448 learned_addr learned my addr 172.21.15.107:0/1372882448 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:01.009 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522ffff700 1 -- 172.21.15.107:0/1372882448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522400b200 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522ffff700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5224006a40 tx=0x7f5224006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5224011c90 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523010d420 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f522401b8c0 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522401a8c0 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.004+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523010d830 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.005+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f522400f070 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.005+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230045ad0 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.005+0000 7f522cff9700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 0x7f521c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.005+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5224040ab0 con 0x7f52300703f0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.005+0000 7f522f7fe700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 0x7f521c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.010 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.006+0000 7f522f7fe700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 0x7f521c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52200096a0 tx=0x7f5220006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.011 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.009+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5224010540 con 0x7f52300703f0 2024-01-21T07:29:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:28:59.812530+0000 mon.smithi107 (mon.0) 81 : cluster [DBG] mgrmap e11: smithi107.adejht(active, since 2s) 2024-01-21T07:29:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:59.914372+0000 mgr.smithi107.adejht (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:28:59.914921+0000 mgr.smithi107.adejht (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T07:29:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:28:59.917103+0000 mon.smithi107 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' 2024-01-21T07:29:01.189 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.187+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 --> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f523010e090 con 0x7f521c02fe60 2024-01-21T07:29:01.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.191+0000 7f522cff9700 1 -- 172.21.15.107:0/1372882448 <== mgr.14142 v2:172.21.15.107:6800/2854824672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f523010e090 con 0x7f521c02fe60 2024-01-21T07:29:01.193 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 msgr2=0x7f521c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 0x7f521c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52200096a0 tx=0x7f5220006b40).stop 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 msgr2=0x7f523010cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5224006a40 tx=0x7f5224006ee0).stop 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 shutdown_connections 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f521c02fe60 0x7f521c032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 --2- 172.21.15.107:0/1372882448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52300703f0 0x7f523010cd30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:01.196 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.193+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 >> 172.21.15.107:0/1372882448 conn(0x7f52300f63c0 msgr2=0x7f52300f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:01.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.194+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 shutdown_connections 2024-01-21T07:29:01.197 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.194+0000 7f523667a700 1 -- 172.21.15.107:0/1372882448 wait complete. 2024-01-21T07:29:01.550 INFO:teuthology.orchestra.run.smithi107.stderr:Deploying alertmanager service with default placement... 2024-01-21T07:29:01.550 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.892+0000 7f0379c3d700 1 Processor -- start 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.892+0000 7f0379c3d700 1 -- start start 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.892+0000 7f0379c3d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f0379c3d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740fb490 con 0x7f03740fab80 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45064/0 (socket says 172.21.15.107:45064) 2024-01-21T07:29:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03737fe700 1 -- 172.21.15.107:0/1118904980 learned_addr learned my addr 172.21.15.107:0/1118904980 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03737fe700 1 -- 172.21.15.107:0/1118904980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740fb5d0 con 0x7f03740fab80 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03737fe700 1 --2- 172.21.15.107:0/1118904980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f036800b5f0 tx=0x7f0368006ee0).ready entity=mon.0 client_cookie=8354d2365e18b7dd server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03727fc700 1 -- 172.21.15.107:0/1118904980 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036800d600 con 0x7f03740fab80 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03727fc700 1 -- 172.21.15.107:0/1118904980 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f036800dc10 con 0x7f03740fab80 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.893+0000 7f03727fc700 1 -- 172.21.15.107:0/1118904980 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0368011cd0 con 0x7f03740fab80 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 -- 172.21.15.107:0/1118904980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 msgr2=0x7f03740fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1118904980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f036800b5f0 tx=0x7f0368006ee0).stop 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 -- 172.21.15.107:0/1118904980 shutdown_connections 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1118904980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f03740fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 -- 172.21.15.107:0/1118904980 >> 172.21.15.107:0/1118904980 conn(0x7f03740f63c0 msgr2=0x7f03740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 -- 172.21.15.107:0/1118904980 shutdown_connections 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.894+0000 7f0379c3d700 1 -- 172.21.15.107:0/1118904980 wait complete. 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f0379c3d700 1 Processor -- start 2024-01-21T07:29:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f0379c3d700 1 -- start start 2024-01-21T07:29:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f0379c3d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f0379c3d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740ff320 con 0x7f03740fab80 2024-01-21T07:29:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f03737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f03737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45074/0 (socket says 172.21.15.107:45074) 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.895+0000 7f03737fe700 1 -- 172.21.15.107:0/1351389900 learned_addr learned my addr 172.21.15.107:0/1351389900 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f03737fe700 1 -- 172.21.15.107:0/1351389900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036800b200 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f03737fe700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f036800dfb0 tx=0x7f0368006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0368011df0 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f036801b8c0 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740ff4c0 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036801a8c0 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.896+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03740ff8d0 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f036800f070 con 0x7f03740fab80 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f035bfff700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 0x7f0360032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:01.901 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0374045ad0 con 0x7f03740fab80 2024-01-21T07:29:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0368041190 con 0x7f03740fab80 2024-01-21T07:29:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f0372ffd700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 0x7f0360032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:01.902 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.897+0000 7f0372ffd700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 0x7f0360032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:01.903 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:01.901+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0368016070 con 0x7f03740fab80 2024-01-21T07:29:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.077+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 --> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f03740ffb10 con 0x7f036002feb0 2024-01-21T07:29:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.080+0000 7f035bfff700 1 -- 172.21.15.107:0/1351389900 <== mgr.14142 v2:172.21.15.107:6800/2854824672 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f03740ffb10 con 0x7f036002feb0 2024-01-21T07:29:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 msgr2=0x7f0360032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 0x7f0360032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).stop 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 msgr2=0x7f0374100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f036800dfb0 tx=0x7f0368006ee0).stop 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 shutdown_connections 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f036002feb0 0x7f0360032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 --2- 172.21.15.107:0/1351389900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03740fab80 0x7f0374100ea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 >> 172.21.15.107:0/1351389900 conn(0x7f03740f63c0 msgr2=0x7f0374070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 shutdown_connections 2024-01-21T07:29:02.086 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.082+0000 7f0379c3d700 1 -- 172.21.15.107:0/1351389900 wait complete. 2024-01-21T07:29:02.440 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T07:29:02.538 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:01.189514+0000 mgr.smithi107.adejht (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:02.538 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:29:01.190144+0000 mgr.smithi107.adejht (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T07:29:02.538 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:01.192466+0000 mon.smithi107 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' 2024-01-21T07:29:02.538 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:02.081671+0000 mon.smithi107 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.107:0/3518143845' entity='mgr.smithi107.adejht' 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.778+0000 7f921158c700 1 Processor -- start 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.778+0000 7f921158c700 1 -- start start 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f921158c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f921158c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c0fc5c0 con 0x7f920c0fbcb0 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f920affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f920affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45090/0 (socket says 172.21.15.107:45090) 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f920affd700 1 -- 172.21.15.107:0/3471550927 learned_addr learned my addr 172.21.15.107:0/3471550927 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f920affd700 1 -- 172.21.15.107:0/3471550927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c0fc700 con 0x7f920c0fbcb0 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f920affd700 1 --2- 172.21.15.107:0/3471550927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f920000b5f0 tx=0x7f9200006ee0).ready entity=mon.0 client_cookie=2f763ac1383e72c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f9209ffb700 1 -- 172.21.15.107:0/3471550927 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920000d600 con 0x7f920c0fbcb0 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f9209ffb700 1 -- 172.21.15.107:0/3471550927 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f920000dc10 con 0x7f920c0fbcb0 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.779+0000 7f9209ffb700 1 -- 172.21.15.107:0/3471550927 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9200011cd0 con 0x7f920c0fbcb0 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 -- 172.21.15.107:0/3471550927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 msgr2=0x7f920c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 --2- 172.21.15.107:0/3471550927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f920000b5f0 tx=0x7f9200006ee0).stop 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 -- 172.21.15.107:0/3471550927 shutdown_connections 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 --2- 172.21.15.107:0/3471550927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c0fc0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 -- 172.21.15.107:0/3471550927 >> 172.21.15.107:0/3471550927 conn(0x7f920c0f74b0 msgr2=0x7f920c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 -- 172.21.15.107:0/3471550927 shutdown_connections 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.780+0000 7f921158c700 1 -- 172.21.15.107:0/3471550927 wait complete. 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f921158c700 1 Processor -- start 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f921158c700 1 -- start start 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f921158c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f921158c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c10af30 con 0x7f920c0fbcb0 2024-01-21T07:29:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f920affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f920affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45094/0 (socket says 172.21.15.107:45094) 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.781+0000 7f920affd700 1 -- 172.21.15.107:0/2959708019 learned_addr learned my addr 172.21.15.107:0/2959708019 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f920affd700 1 -- 172.21.15.107:0/2959708019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920000b200 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f920affd700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f920000e040 tx=0x7f9200006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920001b410 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f920001ba20 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c10b130 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f920001a910 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.782+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c10b540 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f920000f070 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920c045ad0 con 0x7f920c0fbcb0 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f91f37fe700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 0x7f91f8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9200041270 con 0x7f920c0fbcb0 2024-01-21T07:29:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f920a7fc700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 0x7f91f8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.783+0000 7f920a7fc700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 0x7f91f8032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f91fc0096a0 tx=0x7f91fc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.787+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9200019950 con 0x7f920c0fbcb0 2024-01-21T07:29:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.936+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f920c0fda30 con 0x7f920c0fbcb0 2024-01-21T07:29:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.939+0000 7f91f37fe700 1 -- 172.21.15.107:0/2959708019 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f92000160d0 con 0x7f920c0fbcb0 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 msgr2=0x7f91f8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 0x7f91f8032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f91fc0096a0 tx=0x7f91fc006b40).stop 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 msgr2=0x7f920c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f920000e040 tx=0x7f9200006ee0).stop 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 shutdown_connections 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f91f802fe60 0x7f91f8032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:02.944 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 --2- 172.21.15.107:0/2959708019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f920c0fbcb0 0x7f920c10aa40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:02.945 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 >> 172.21.15.107:0/2959708019 conn(0x7f920c0f74b0 msgr2=0x7f920c0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:02.945 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 shutdown_connections 2024-01-21T07:29:02.945 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:02.941+0000 7f921158c700 1 -- 172.21.15.107:0/2959708019 wait complete. 2024-01-21T07:29:03.234 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:02.078882+0000 mgr.smithi107.adejht (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:03.235 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:29:02.079389+0000 mgr.smithi107.adejht (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T07:29:03.235 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:02.940626+0000 mon.smithi107 (mon.0) 85 : audit [INF] from='client.? 172.21.15.107:0/2959708019' entity='client.admin' 2024-01-21T07:29:03.291 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T07:29:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.653+0000 7fdc41093700 1 Processor -- start 2024-01-21T07:29:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.653+0000 7fdc41093700 1 -- start start 2024-01-21T07:29:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.653+0000 7fdc41093700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.653+0000 7fdc41093700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c0fc6d0 con 0x7fdc3c070430 2024-01-21T07:29:03.660 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc3ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc3ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45102/0 (socket says 172.21.15.107:45102) 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc3ad9d700 1 -- 172.21.15.107:0/3370315563 learned_addr learned my addr 172.21.15.107:0/3370315563 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc3ad9d700 1 -- 172.21.15.107:0/3370315563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c1011d0 con 0x7fdc3c070430 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc3ad9d700 1 --2- 172.21.15.107:0/3370315563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdc3000b5f0 tx=0x7fdc30006ee0).ready entity=mon.0 client_cookie=60d3838f3977dfe6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc39d9b700 1 -- 172.21.15.107:0/3370315563 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3000d600 con 0x7fdc3c070430 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc39d9b700 1 -- 172.21.15.107:0/3370315563 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3000dc10 con 0x7fdc3c070430 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.654+0000 7fdc39d9b700 1 -- 172.21.15.107:0/3370315563 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc30011cd0 con 0x7fdc3c070430 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 -- 172.21.15.107:0/3370315563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 msgr2=0x7fdc3c100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 --2- 172.21.15.107:0/3370315563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdc3000b5f0 tx=0x7fdc30006ee0).stop 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 -- 172.21.15.107:0/3370315563 shutdown_connections 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 --2- 172.21.15.107:0/3370315563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c100ce0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 -- 172.21.15.107:0/3370315563 >> 172.21.15.107:0/3370315563 conn(0x7fdc3c0f63c0 msgr2=0x7fdc3c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 -- 172.21.15.107:0/3370315563 shutdown_connections 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.655+0000 7fdc41093700 1 -- 172.21.15.107:0/3370315563 wait complete. 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc41093700 1 Processor -- start 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc41093700 1 -- start start 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc41093700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc41093700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c10d220 con 0x7fdc3c070430 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc3ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc3ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45106/0 (socket says 172.21.15.107:45106) 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc3ad9d700 1 -- 172.21.15.107:0/3415170214 learned_addr learned my addr 172.21.15.107:0/3415170214 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc3ad9d700 1 -- 172.21.15.107:0/3415170214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3000b200 con 0x7fdc3c070430 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc3ad9d700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdc3000df90 tx=0x7fdc30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.656+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc30011da0 con 0x7fdc3c070430 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3001b8c0 con 0x7fdc3c070430 2024-01-21T07:29:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c10d420 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3001a930 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c10d830 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fdc3000f070 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.657+0000 7fdc237fe700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 0x7fdc28032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.658+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdc30041100 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.658+0000 7fdc3a59c700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 0x7fdc28032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.658+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c045ad0 con 0x7fdc3c070430 2024-01-21T07:29:03.663 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.658+0000 7fdc3a59c700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 0x7fdc28032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc2c0096a0 tx=0x7fdc2c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:03.664 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.662+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc30016070 con 0x7fdc3c070430 2024-01-21T07:29:03.815 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.813+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fdc3c106b40 con 0x7fdc3c070430 2024-01-21T07:29:03.818 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.816+0000 7fdc237fe700 1 -- 172.21.15.107:0/3415170214 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fdc30010be0 con 0x7fdc3c070430 2024-01-21T07:29:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 msgr2=0x7fdc28032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 0x7fdc28032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdc2c0096a0 tx=0x7fdc2c006b40).stop 2024-01-21T07:29:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 msgr2=0x7fdc3c10cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdc3000df90 tx=0x7fdc30006ee0).stop 2024-01-21T07:29:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 shutdown_connections 2024-01-21T07:29:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7fdc2802fe60 0x7fdc28032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 --2- 172.21.15.107:0/3415170214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc3c070430 0x7fdc3c10cd30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 >> 172.21.15.107:0/3415170214 conn(0x7fdc3c0f63c0 msgr2=0x7fdc3c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 shutdown_connections 2024-01-21T07:29:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:03.818+0000 7fdc41093700 1 -- 172.21.15.107:0/3415170214 wait complete. 2024-01-21T07:29:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:Enabling the dashboard module... 2024-01-21T07:29:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T07:29:04.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.539+0000 7f23ee1d2700 1 Processor -- start 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.539+0000 7f23ee1d2700 1 -- start start 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.539+0000 7f23ee1d2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.539+0000 7f23ee1d2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80728d0 con 0x7f23e80724b0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45108/0 (socket says 172.21.15.107:45108) 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e77fe700 1 -- 172.21.15.107:0/1081122227 learned_addr learned my addr 172.21.15.107:0/1081122227 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e77fe700 1 -- 172.21.15.107:0/1081122227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8070f40 con 0x7f23e80724b0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e77fe700 1 --2- 172.21.15.107:0/1081122227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f23dc00b5f0 tx=0x7f23dc006ee0).ready entity=mon.0 client_cookie=ef8bdf5351ba107d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e67fc700 1 -- 172.21.15.107:0/1081122227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23dc00d600 con 0x7f23e80724b0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e67fc700 1 -- 172.21.15.107:0/1081122227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23dc00dc10 con 0x7f23e80724b0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.540+0000 7f23e67fc700 1 -- 172.21.15.107:0/1081122227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23dc011cd0 con 0x7f23e80724b0 2024-01-21T07:29:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 -- 172.21.15.107:0/1081122227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 msgr2=0x7f23e8070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/1081122227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f23dc00b5f0 tx=0x7f23dc006ee0).stop 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 -- 172.21.15.107:0/1081122227 shutdown_connections 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/1081122227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e8070a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 -- 172.21.15.107:0/1081122227 >> 172.21.15.107:0/1081122227 conn(0x7f23e80f74c0 msgr2=0x7f23e80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 -- 172.21.15.107:0/1081122227 shutdown_connections 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.541+0000 7f23ee1d2700 1 -- 172.21.15.107:0/1081122227 wait complete. 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23ee1d2700 1 Processor -- start 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23ee1d2700 1 -- start start 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23ee1d2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23ee1d2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e810d200 con 0x7f23e80724b0 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45114/0 (socket says 172.21.15.107:45114) 2024-01-21T07:29:04.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.542+0000 7f23e77fe700 1 -- 172.21.15.107:0/854120572 learned_addr learned my addr 172.21.15.107:0/854120572 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23e77fe700 1 -- 172.21.15.107:0/854120572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc00b200 con 0x7f23e80724b0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23e77fe700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f23dc00b1d0 tx=0x7f23dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23dc011df0 con 0x7f23e80724b0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e810d400 con 0x7f23e80724b0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23dc01b8c0 con 0x7f23e80724b0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23dc01a830 con 0x7f23e80724b0 2024-01-21T07:29:04.548 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.543+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e810d810 con 0x7f23e80724b0 2024-01-21T07:29:04.549 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f23dc00f070 con 0x7f23e80724b0 2024-01-21T07:29:04.549 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e8045ad0 con 0x7f23e80724b0 2024-01-21T07:29:04.549 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23cffff700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 0x7f23d4032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:04.549 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f23dc040fa0 con 0x7f23e80724b0 2024-01-21T07:29:04.550 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23e6ffd700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 0x7f23d4032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:04.550 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.544+0000 7f23e6ffd700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 0x7f23d4032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f23d80096a0 tx=0x7f23d8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:04.550 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.548+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23dc019d00 con 0x7f23e80724b0 2024-01-21T07:29:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.733+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f23e8101e20 con 0x7f23e80724b0 2024-01-21T07:29:04.824 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.821+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f23dc01baf0 con 0x7f23e80724b0 2024-01-21T07:29:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.822+0000 7f23cffff700 1 -- 172.21.15.107:0/854120572 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f23dc0237e0 con 0x7f23e80724b0 2024-01-21T07:29:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 msgr2=0x7f23d4032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 0x7f23d4032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f23d80096a0 tx=0x7f23d8006b40).stop 2024-01-21T07:29:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 msgr2=0x7f23e810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f23dc00b1d0 tx=0x7f23dc006ee0).stop 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 shutdown_connections 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f23d402fe60 0x7f23d4032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 --2- 172.21.15.107:0/854120572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23e80724b0 0x7f23e810cd10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 >> 172.21.15.107:0/854120572 conn(0x7f23e80f74c0 msgr2=0x7f23e80f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 shutdown_connections 2024-01-21T07:29:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:04.824+0000 7f23ee1d2700 1 -- 172.21.15.107:0/854120572 wait complete. 2024-01-21T07:29:05.101 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:03.817579+0000 mon.smithi107 (mon.0) 86 : audit [INF] from='client.? 172.21.15.107:0/3415170214' entity='client.admin' 2024-01-21T07:29:05.101 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:04.735088+0000 mon.smithi107 (mon.0) 87 : audit [INF] from='client.? 172.21.15.107:0/854120572' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T07:29:05.182 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.515+0000 7f6a31d13700 1 Processor -- start 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.515+0000 7f6a31d13700 1 -- start start 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.515+0000 7f6a31d13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.515+0000 7f6a31d13700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0fee10 con 0x7f6a2c0703f0 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.515+0000 7f6a2b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45142/0 (socket says 172.21.15.107:45142) 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2b7fe700 1 -- 172.21.15.107:0/4232427447 learned_addr learned my addr 172.21.15.107:0/4232427447 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:05.522 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2b7fe700 1 -- 172.21.15.107:0/4232427447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0fef50 con 0x7f6a2c0703f0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2b7fe700 1 --2- 172.21.15.107:0/4232427447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6a1c00b5f0 tx=0x7f6a1c006ee0).ready entity=mon.0 client_cookie=b960733cf0dc51f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2a7fc700 1 -- 172.21.15.107:0/4232427447 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a1c00d600 con 0x7f6a2c0703f0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2a7fc700 1 -- 172.21.15.107:0/4232427447 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a1c00dc10 con 0x7f6a2c0703f0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.516+0000 7f6a2a7fc700 1 -- 172.21.15.107:0/4232427447 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a1c011cd0 con 0x7f6a2c0703f0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- 172.21.15.107:0/4232427447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 msgr2=0x7f6a2c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 --2- 172.21.15.107:0/4232427447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6a1c00b5f0 tx=0x7f6a1c006ee0).stop 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- 172.21.15.107:0/4232427447 shutdown_connections 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 --2- 172.21.15.107:0/4232427447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fe920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- 172.21.15.107:0/4232427447 >> 172.21.15.107:0/4232427447 conn(0x7f6a2c0f63c0 msgr2=0x7f6a2c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- 172.21.15.107:0/4232427447 shutdown_connections 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- 172.21.15.107:0/4232427447 wait complete. 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 Processor -- start 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.517+0000 7f6a31d13700 1 -- start start 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a31d13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a31d13700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0fa820 con 0x7f6a2c0703f0 2024-01-21T07:29:05.523 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a2b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a2b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45154/0 (socket says 172.21.15.107:45154) 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a2b7fe700 1 -- 172.21.15.107:0/3105000166 learned_addr learned my addr 172.21.15.107:0/3105000166 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a2b7fe700 1 -- 172.21.15.107:0/3105000166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c00b200 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a2b7fe700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6a1c006a40 tx=0x7f6a1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a1c011c90 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c0fa9c0 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a1c01b8c0 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a1c01a8c0 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.518+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c0fadd0 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.519+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6a1c00f070 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.519+0000 7f6a13fff700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 0x7f6a18032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.519+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6a1c041a00 con 0x7f6a2c0703f0 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.519+0000 7f6a2affd700 1 -- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 msgr2=0x7f6a18032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:05.524 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.519+0000 7f6a2affd700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 0x7f6a18032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T07:29:05.525 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.520+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c059100 con 0x7f6a2c0703f0 2024-01-21T07:29:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.524+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a1c016070 con 0x7f6a2c0703f0 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.718+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f6a2c104860 con 0x7f6a2c0703f0 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.718+0000 7f6a13fff700 1 -- 172.21.15.107:0/3105000166 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f6a1c010540 con 0x7f6a2c0703f0 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "epoch": 12, 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "available": true, 2024-01-21T07:29:05.721 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "active_name": "smithi107.adejht", 2024-01-21T07:29:05.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T07:29:05.722 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:29:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a2affd700 1 -- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 msgr2=0x7f6a18032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:05.723 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a2affd700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 0x7f6a18032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 msgr2=0x7f6a18032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 0x7f6a18032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 msgr2=0x7f6a2c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6a1c006a40 tx=0x7f6a1c006ee0).stop 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 shutdown_connections 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f6a1802feb0 0x7f6a18032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 --2- 172.21.15.107:0/3105000166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a2c0703f0 0x7f6a2c0fc3a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 >> 172.21.15.107:0/3105000166 conn(0x7f6a2c0f63c0 msgr2=0x7f6a2c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 shutdown_connections 2024-01-21T07:29:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:05.720+0000 7f6a31d13700 1 -- 172.21.15.107:0/3105000166 wait complete. 2024-01-21T07:29:06.009 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:04.823867+0000 mon.smithi107 (mon.0) 88 : audit [INF] from='client.? 172.21.15.107:0/854120572' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T07:29:06.009 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:04.823934+0000 mon.smithi107 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi107.adejht(active, since 7s) 2024-01-21T07:29:06.009 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:05.719930+0000 mon.smithi107 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.107:0/3105000166' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T07:29:06.092 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for the mgr to restart... 2024-01-21T07:29:06.092 INFO:teuthology.orchestra.run.smithi107.stderr:Waiting for mgr epoch 12... 2024-01-21T07:29:06.092 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T07:29:06.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1f30c700 1 Processor -- start 2024-01-21T07:29:06.464 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1f30c700 1 -- start start 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1f30c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1f30c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d180fb490 con 0x7f0d180fab80 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1d0a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1d0a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45166/0 (socket says 172.21.15.107:45166) 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.461+0000 7f0d1d0a8700 1 -- 172.21.15.107:0/1895206045 learned_addr learned my addr 172.21.15.107:0/1895206045 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.462+0000 7f0d1d0a8700 1 -- 172.21.15.107:0/1895206045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d180fb5d0 con 0x7f0d180fab80 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.462+0000 7f0d1d0a8700 1 --2- 172.21.15.107:0/1895206045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0d1400b870 tx=0x7f0d14006f70).ready entity=mon.0 client_cookie=f1d8022b7190ad48 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.462+0000 7f0d07fff700 1 -- 172.21.15.107:0/1895206045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d1400da20 con 0x7f0d180fab80 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.462+0000 7f0d07fff700 1 -- 172.21.15.107:0/1895206045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d14010470 con 0x7f0d180fab80 2024-01-21T07:29:06.465 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.462+0000 7f0d07fff700 1 -- 172.21.15.107:0/1895206045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d1400f5b0 con 0x7f0d180fab80 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1895206045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 msgr2=0x7f0d180fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1895206045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0d1400b870 tx=0x7f0d14006f70).stop 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1895206045 shutdown_connections 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1895206045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d180fafa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1895206045 >> 172.21.15.107:0/1895206045 conn(0x7f0d180f63c0 msgr2=0x7f0d180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1895206045 shutdown_connections 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1895206045 wait complete. 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 Processor -- start 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.463+0000 7f0d1f30c700 1 -- start start 2024-01-21T07:29:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1f30c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1f30c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1810d1c0 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1d0a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1d0a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45170/0 (socket says 172.21.15.107:45170) 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1d0a8700 1 -- 172.21.15.107:0/1079313088 learned_addr learned my addr 172.21.15.107:0/1079313088 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1d0a8700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1400b480 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1d0a8700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0d1400b840 tx=0x7f0d14006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d1400de60 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d1400fef0 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1810d3c0 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d14019de0 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.464+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1810d7d0 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.465+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0d14016030 con 0x7f0d180fab80 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.465+0000 7f0d05ffb700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.465+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d00032a30 con 0x7f0d0002feb0 2024-01-21T07:29:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.466+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0d14041fd0 con 0x7f0d180fab80 2024-01-21T07:29:06.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.466+0000 7f0d1c8a7700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:06.470 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.466+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T07:29:06.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.667+0000 7f0d1c8a7700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:06.669 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:06.667+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T07:29:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:07.068+0000 7f0d1c8a7700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:07.070 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:07.068+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T07:29:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:07.869+0000 7f0d1c8a7700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:07.869+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T07:29:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:09.470+0000 7f0d1c8a7700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.107:6800/2854824672 2024-01-21T07:29:09.473 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:09.470+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T07:29:09.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:09.805+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f0d14023ce0 con 0x7f0d180fab80 2024-01-21T07:29:09.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:09.806+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 msgr2=0x7f0d00032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T07:29:09.808 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:09.806+0000 7f0d05ffb700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:29:10.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:09.755177+0000 mon.smithi107 (mon.0) 91 : cluster [INF] Active manager daemon smithi107.adejht restarted 2024-01-21T07:29:10.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:09.755567+0000 mon.smithi107 (mon.0) 92 : cluster [INF] Activating manager daemon smithi107.adejht 2024-01-21T07:29:10.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:09.757253+0000 mon.smithi107 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T07:29:10.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.817+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f0d140434b0 con 0x7f0d180fab80 2024-01-21T07:29:10.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.817+0000 7f0d05ffb700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 0x7f0d00035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:10.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.817+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d00032a30 con 0x7f0d00033b40 2024-01-21T07:29:10.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.818+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 0x7f0d00035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:10.821 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.818+0000 7f0d1c8a7700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 0x7f0d00035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0d08006fd0 tx=0x7f0d08006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:10.822 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.819+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f0d00032a30 con 0x7f0d00033b40 2024-01-21T07:29:10.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0d18059100 con 0x7f0d00033b40 2024-01-21T07:29:10.827 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d05ffb700 1 -- 172.21.15.107:0/1079313088 <== mgr.14162 v2:172.21.15.107:6800/1892560072 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0d18059100 con 0x7f0d00033b40 2024-01-21T07:29:10.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: { 2024-01-21T07:29:10.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2024-01-21T07:29:10.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T07:29:10.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: } 2024-01-21T07:29:10.828 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 msgr2=0x7f0d00035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 0x7f0d00035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0d08006fd0 tx=0x7f0d08006cf0).stop 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 msgr2=0x7f0d1810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.825+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f0d1400b840 tx=0x7f0d14006f70).stop 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 shutdown_connections 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d00033b40 0x7f0d00035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d180fab80 0x7f0d1810ccd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 --2- 172.21.15.107:0/1079313088 >> [v2:172.21.15.107:6800/2854824672,v1:172.21.15.107:6801/2854824672] conn(0x7f0d0002feb0 0x7f0d00032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 >> 172.21.15.107:0/1079313088 conn(0x7f0d180f63c0 msgr2=0x7f0d180ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 shutdown_connections 2024-01-21T07:29:10.829 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:10.826+0000 7f0d1f30c700 1 -- 172.21.15.107:0/1079313088 wait complete. 2024-01-21T07:29:10.996 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:09.807877+0000 mon.smithi107 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi107.adejht(active, starting, since 0.0523892s) 2024-01-21T07:29:10.996 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.809580+0000 mon.smithi107 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata", "id": "smithi107"}]: dispatch 2024-01-21T07:29:10.996 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.810146+0000 mon.smithi107 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mgr metadata", "who": "smithi107.adejht", "id": "smithi107.adejht"}]: dispatch 2024-01-21T07:29:10.996 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.810766+0000 mon.smithi107 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.810907+0000 mon.smithi107 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.811031+0000 mon.smithi107 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:09.824615+0000 mon.smithi107 (mon.0) 100 : cluster [INF] Manager daemon smithi107.adejht is now available 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.925353+0000 mon.smithi107 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.927750+0000 mon.smithi107 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.943340+0000 mon.smithi107 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:09.945973+0000 mon.smithi107 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:29:10.997 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:10.766528+0000 mon.smithi107 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:mgr epoch 12 is available 2024-01-21T07:29:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:Generating a dashboard self-signed certificate... 2024-01-21T07:29:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T07:29:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f51992de700 1 Processor -- start 2024-01-21T07:29:11.496 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f51992de700 1 -- start start 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f51992de700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f51992de700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940fb2e0 con 0x7f51940fa9d0 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f5193fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f5193fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57506/0 (socket says 172.21.15.107:57506) 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f5193fff700 1 -- 172.21.15.107:0/1691145539 learned_addr learned my addr 172.21.15.107:0/1691145539 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f5193fff700 1 -- 172.21.15.107:0/1691145539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940fb420 con 0x7f51940fa9d0 2024-01-21T07:29:11.497 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.493+0000 7f5193fff700 1 --2- 172.21.15.107:0/1691145539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f518400b5f0 tx=0x7f5184006ee0).ready entity=mon.0 client_cookie=a129e51a065b3a05 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f5192ffd700 1 -- 172.21.15.107:0/1691145539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518400d600 con 0x7f51940fa9d0 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f5192ffd700 1 -- 172.21.15.107:0/1691145539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f518400dc10 con 0x7f51940fa9d0 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f5192ffd700 1 -- 172.21.15.107:0/1691145539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5184011cd0 con 0x7f51940fa9d0 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 -- 172.21.15.107:0/1691145539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 msgr2=0x7f51940fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 --2- 172.21.15.107:0/1691145539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f518400b5f0 tx=0x7f5184006ee0).stop 2024-01-21T07:29:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 -- 172.21.15.107:0/1691145539 shutdown_connections 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 --2- 172.21.15.107:0/1691145539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f51940fadf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 -- 172.21.15.107:0/1691145539 >> 172.21.15.107:0/1691145539 conn(0x7f51940f6210 msgr2=0x7f51940f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 -- 172.21.15.107:0/1691145539 shutdown_connections 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.494+0000 7f51992de700 1 -- 172.21.15.107:0/1691145539 wait complete. 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f51992de700 1 Processor -- start 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f51992de700 1 -- start start 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f51992de700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f51992de700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519410cf60 con 0x7f51940fa9d0 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f5193fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f5193fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57512/0 (socket says 172.21.15.107:57512) 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f5193fff700 1 -- 172.21.15.107:0/636328653 learned_addr learned my addr 172.21.15.107:0/636328653 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f5193fff700 1 -- 172.21.15.107:0/636328653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518400b200 con 0x7f51940fa9d0 2024-01-21T07:29:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.495+0000 7f5193fff700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f518400e040 tx=0x7f5184006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518401b410 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f518401ba20 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519410d160 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518401a910 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519410d570 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.496+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f518400f070 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.497+0000 7f5190ff9700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 0x7f5180032250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.497+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51840410d0 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.497+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5194045ad0 con 0x7f51940fa9d0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.497+0000 7f51937fe700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 0x7f5180032250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:11.501 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.497+0000 7f51937fe700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 0x7f5180032250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f51880096a0 tx=0x7f5188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:11.503 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.501+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5184016070 con 0x7f51940fa9d0 2024-01-21T07:29:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.672+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f51941024c0 con 0x7f518002fd90 2024-01-21T07:29:11.747 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.745+0000 7f5190ff9700 1 -- 172.21.15.107:0/636328653 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f51941024c0 con 0x7f518002fd90 2024-01-21T07:29:11.747 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 msgr2=0x7f5180032250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 0x7f5180032250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f51880096a0 tx=0x7f5188006b40).stop 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 msgr2=0x7f519410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f518400e040 tx=0x7f5184006ee0).stop 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 shutdown_connections 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f518002fd90 0x7f5180032250 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 --2- 172.21.15.107:0/636328653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51940fa9d0 0x7f519410ca70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 >> 172.21.15.107:0/636328653 conn(0x7f51940f6210 msgr2=0x7f51940ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 shutdown_connections 2024-01-21T07:29:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:11.747+0000 7f51992de700 1 -- 172.21.15.107:0/636328653 wait complete. 2024-01-21T07:29:12.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:10.820129+0000 mgr.smithi107.adejht (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T07:29:12.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:10.820554+0000 mon.smithi107 (mon.0) 106 : cluster [DBG] mgrmap e14: smithi107.adejht(active, since 1.06506s) 2024-01-21T07:29:12.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:10.826453+0000 mgr.smithi107.adejht (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T07:29:12.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:11.743246+0000 mon.smithi107 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:12.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:11.746347+0000 mon.smithi107 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:12.130 INFO:teuthology.orchestra.run.smithi107.stderr:Creating initial admin user... 2024-01-21T07:29:12.131 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmppcuy66ff:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.536+0000 7f8f4a45c700 1 Processor -- start 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.536+0000 7f8f4a45c700 1 -- start start 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.536+0000 7f8f4a45c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.536+0000 7f8f4a45c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440fc730 con 0x7f8f4406fbb0 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.536+0000 7f8f43fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f43fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57524/0 (socket says 172.21.15.107:57524) 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f43fff700 1 -- 172.21.15.107:0/886305900 learned_addr learned my addr 172.21.15.107:0/886305900 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f43fff700 1 -- 172.21.15.107:0/886305900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44101230 con 0x7f8f4406fbb0 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f43fff700 1 --2- 172.21.15.107:0/886305900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8f3800b5f0 tx=0x7f8f38006ee0).ready entity=mon.0 client_cookie=272f79389ec4f4d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f42ffd700 1 -- 172.21.15.107:0/886305900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3800d600 con 0x7f8f4406fbb0 2024-01-21T07:29:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f42ffd700 1 -- 172.21.15.107:0/886305900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f3800dc10 con 0x7f8f4406fbb0 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.537+0000 7f8f42ffd700 1 -- 172.21.15.107:0/886305900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38011cd0 con 0x7f8f4406fbb0 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 -- 172.21.15.107:0/886305900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 msgr2=0x7f8f44100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/886305900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8f3800b5f0 tx=0x7f8f38006ee0).stop 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 -- 172.21.15.107:0/886305900 shutdown_connections 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/886305900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f44100d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 -- 172.21.15.107:0/886305900 >> 172.21.15.107:0/886305900 conn(0x7f8f440f63a0 msgr2=0x7f8f440f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 -- 172.21.15.107:0/886305900 shutdown_connections 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 -- 172.21.15.107:0/886305900 wait complete. 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.538+0000 7f8f4a45c700 1 Processor -- start 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f4a45c700 1 -- start start 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f4a45c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f4a45c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4410d220 con 0x7f8f4406fbb0 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f43fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f43fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57534/0 (socket says 172.21.15.107:57534) 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f43fff700 1 -- 172.21.15.107:0/1334979470 learned_addr learned my addr 172.21.15.107:0/1334979470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f43fff700 1 -- 172.21.15.107:0/1334979470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3800b200 con 0x7f8f4406fbb0 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f43fff700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8f38000c00 tx=0x7f8f38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38011df0 con 0x7f8f4406fbb0 2024-01-21T07:29:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4410d420 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f3801b8c0 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3801a6e0 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.539+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4410d830 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.540+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f8f3802b070 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.540+0000 7f8f40ff9700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 0x7f8f30032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.540+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f38041650 con 0x7f8f4406fbb0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.540+0000 7f8f437fe700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 0x7f8f30032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.541+0000 7f8f437fe700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 0x7f8f30032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8f340096a0 tx=0x7f8f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.541+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f44045ad0 con 0x7f8f4406fbb0 2024-01-21T07:29:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.545+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f3801f020 con 0x7f8f4406fbb0 2024-01-21T07:29:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.711+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8f44101e00 con 0x7f8f3002fdb0 2024-01-21T07:29:12.748 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.745+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8f3800f070 con 0x7f8f4406fbb0 2024-01-21T07:29:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.930+0000 7f8f40ff9700 1 -- 172.21.15.107:0/1334979470 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f8f44101e00 con 0x7f8f3002fdb0 2024-01-21T07:29:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$9EaUPbE6/IM0I3zzPBnw.ucBnNNSftr32C2YJqhVZQTn28xkoEIjC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705822152, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.931+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 msgr2=0x7f8f30032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.931+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 0x7f8f30032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8f340096a0 tx=0x7f8f34006b40).stop 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.931+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 msgr2=0x7f8f4410cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.931+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8f38000c00 tx=0x7f8f38006ee0).stop 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 shutdown_connections 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8f3002fdb0 0x7f8f30032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 --2- 172.21.15.107:0/1334979470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f4406fbb0 0x7f8f4410cd30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 >> 172.21.15.107:0/1334979470 conn(0x7f8f440f63a0 msgr2=0x7f8f440f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 shutdown_connections 2024-01-21T07:29:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:12.932+0000 7f8f4a45c700 1 -- 172.21.15.107:0/1334979470 wait complete. 2024-01-21T07:29:13.096 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:11.674447+0000 mgr.smithi107.adejht (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:13.096 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:12.748327+0000 mon.smithi107 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi107.adejht(active, since 2s) 2024-01-21T07:29:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:Fetching dashboard port number... 2024-01-21T07:29:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9v9j9ro4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcnfa6skr:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.624+0000 7f462700f700 1 Processor -- start 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.624+0000 7f462700f700 1 -- start start 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f462700f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f462700f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200fedd0 con 0x7f4620070430 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f4624dab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f4624dab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57546/0 (socket says 172.21.15.107:57546) 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f4624dab700 1 -- 172.21.15.107:0/3673204583 learned_addr learned my addr 172.21.15.107:0/3673204583 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f4624dab700 1 -- 172.21.15.107:0/3673204583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46200fef10 con 0x7f4620070430 2024-01-21T07:29:13.629 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f4624dab700 1 --2- 172.21.15.107:0/3673204583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f461400b5f0 tx=0x7f4614006ee0).ready entity=mon.0 client_cookie=3fbfe617a0378fa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f461f7fe700 1 -- 172.21.15.107:0/3673204583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f461400d600 con 0x7f4620070430 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f461f7fe700 1 -- 172.21.15.107:0/3673204583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f461400dc10 con 0x7f4620070430 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.625+0000 7f461f7fe700 1 -- 172.21.15.107:0/3673204583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4614011cd0 con 0x7f4620070430 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 -- 172.21.15.107:0/3673204583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 msgr2=0x7f46200fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 --2- 172.21.15.107:0/3673204583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f461400b5f0 tx=0x7f4614006ee0).stop 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 -- 172.21.15.107:0/3673204583 shutdown_connections 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 --2- 172.21.15.107:0/3673204583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f46200fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 -- 172.21.15.107:0/3673204583 >> 172.21.15.107:0/3673204583 conn(0x7f46200f63c0 msgr2=0x7f46200f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 -- 172.21.15.107:0/3673204583 shutdown_connections 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.626+0000 7f462700f700 1 -- 172.21.15.107:0/3673204583 wait complete. 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f462700f700 1 Processor -- start 2024-01-21T07:29:13.630 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f462700f700 1 -- start start 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f462700f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f462700f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462010afd0 con 0x7f4620070430 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f4624dab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f4624dab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57562/0 (socket says 172.21.15.107:57562) 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f4624dab700 1 -- 172.21.15.107:0/2965986515 learned_addr learned my addr 172.21.15.107:0/2965986515 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.627+0000 7f4624dab700 1 -- 172.21.15.107:0/2965986515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461400b200 con 0x7f4620070430 2024-01-21T07:29:13.632 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f4624dab700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4614006800 tx=0x7f4614006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4614011df0 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462010b1d0 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f461401b8c0 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f461401a830 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462010b5e0 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.628+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f461400f070 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.629+0000 7f461d7fa700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 0x7f4604032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.629+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4614041fe0 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.629+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620059100 con 0x7f4620070430 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.629+0000 7f461ffff700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 0x7f4604032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:13.633 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.629+0000 7f461ffff700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 0x7f4604032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f460c0096a0 tx=0x7f460c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:13.635 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.633+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4614016070 con 0x7f4620070430 2024-01-21T07:29:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.783+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4620045ad0 con 0x7f4620070430 2024-01-21T07:29:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.784+0000 7f461d7fa700 1 -- 172.21.15.107:0/2965986515 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4614024410 con 0x7f4620070430 2024-01-21T07:29:13.786 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 8443 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.785+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 msgr2=0x7f4604032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.785+0000 7f462700f700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 0x7f4604032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f460c0096a0 tx=0x7f460c006b40).stop 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.785+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 msgr2=0x7f462010aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.785+0000 7f462700f700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4614006800 tx=0x7f4614006ee0).stop 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 shutdown_connections 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f460402feb0 0x7f4604032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 --2- 172.21.15.107:0/2965986515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4620070430 0x7f462010aae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 >> 172.21.15.107:0/2965986515 conn(0x7f46200f63c0 msgr2=0x7f46200f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 shutdown_connections 2024-01-21T07:29:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:/usr/bin/ceph: 2024-01-21T07:29:13.786+0000 7f462700f700 1 -- 172.21.15.107:0/2965986515 wait complete. 2024-01-21T07:29:14.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:12.713497+0000 mgr.smithi107.adejht (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:14.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:12.931226+0000 mon.smithi107 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:14.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:13.785159+0000 mon.smithi107 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.107:0/2965986515' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T07:29:14.130 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T07:29:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: disabled 2024-01-21T07:29:14.138 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T07:29:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:systemctl: inactive 2024-01-21T07:29:14.145 INFO:teuthology.orchestra.run.smithi107.stderr:firewalld.service is not enabled 2024-01-21T07:29:14.146 INFO:teuthology.orchestra.run.smithi107.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr:Ceph Dashboard is now available at: 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: User: admin 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: Password: g2tdiionfv 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr:You can access the Ceph CLI with: 2024-01-21T07:29:14.148 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: ceph telemetry on 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr:For more information see: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr:Bootstrap complete. 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr:Releasing lock 140162507057640 on /run/cephadm/962f5a9a-b82e-11ee-95b0-87774f69a715.lock 2024-01-21T07:29:14.149 INFO:teuthology.orchestra.run.smithi107.stderr:Lock 140162507057640 released on /run/cephadm/962f5a9a-b82e-11ee-95b0-87774f69a715.lock 2024-01-21T07:29:14.181 INFO:tasks.cephadm:Fetching config... 2024-01-21T07:29:14.181 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:29:14.181 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T07:29:14.200 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T07:29:14.200 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:29:14.200 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T07:29:14.258 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T07:29:14.258 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:29:14.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/mon.smithi107/keyring of=/dev/stdout 2024-01-21T07:29:14.325 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T07:29:14.325 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-01-21T07:29:14.325 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T07:29:14.383 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T07:29:14.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgOZD2X3LgS9GyqBgGlKKvA5x6UEkHQzBDs/G8WLLKOXtScyneh9bUbz8t5i9ByAG8+X1xsWFSqwfhTCdvEaq1ucHCVhN6D9fQVdQbtSY/CDpxy+hQIfL+gOfFKaeSAH8a59S1gqAX455MS7T0Ob8c+2ut9z7LXCynuKf68WqYbcSmyQ8ez8SR25/l49nxlngV/FNUONVj5N3V3jAMxGvV32/DNXfsMO5ldcH8ALEpdXK+1DCLLSBIE3KrdRMas7hIj1gebuzg7RjMoASDaU+ygqPd7wtLtdquHVDyIkGMOhWyWuMeG9Febk9LLdORwlhGGjY80aFFGySK+4t+PPx0mpF5yixwR2UeWG8vbh0oYNDZ4c7GsAylqdR5VIawkww4E+aRYIg+PgutJ6r9UX9Dd17BL35bitWeg7hUHwgLFuFIBZF72ITpY7c0QsFFfkORgzGZuPGCZnSnpZuds108iAk2W4X6h8zH1QeeRs7/5Cdg52HHCoEwwm8g7jvw2iM= ceph-962f5a9a-b82e-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T07:29:14.460 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgOZD2X3LgS9GyqBgGlKKvA5x6UEkHQzBDs/G8WLLKOXtScyneh9bUbz8t5i9ByAG8+X1xsWFSqwfhTCdvEaq1ucHCVhN6D9fQVdQbtSY/CDpxy+hQIfL+gOfFKaeSAH8a59S1gqAX455MS7T0Ob8c+2ut9z7LXCynuKf68WqYbcSmyQ8ez8SR25/l49nxlngV/FNUONVj5N3V3jAMxGvV32/DNXfsMO5ldcH8ALEpdXK+1DCLLSBIE3KrdRMas7hIj1gebuzg7RjMoASDaU+ygqPd7wtLtdquHVDyIkGMOhWyWuMeG9Febk9LLdORwlhGGjY80aFFGySK+4t+PPx0mpF5yixwR2UeWG8vbh0oYNDZ4c7GsAylqdR5VIawkww4E+aRYIg+PgutJ6r9UX9Dd17BL35bitWeg7hUHwgLFuFIBZF72ITpY7c0QsFFfkORgzGZuPGCZnSnpZuds108iAk2W4X6h8zH1QeeRs7/5Cdg52HHCoEwwm8g7jvw2iM= ceph-962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:29:14.470 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgOZD2X3LgS9GyqBgGlKKvA5x6UEkHQzBDs/G8WLLKOXtScyneh9bUbz8t5i9ByAG8+X1xsWFSqwfhTCdvEaq1ucHCVhN6D9fQVdQbtSY/CDpxy+hQIfL+gOfFKaeSAH8a59S1gqAX455MS7T0Ob8c+2ut9z7LXCynuKf68WqYbcSmyQ8ez8SR25/l49nxlngV/FNUONVj5N3V3jAMxGvV32/DNXfsMO5ldcH8ALEpdXK+1DCLLSBIE3KrdRMas7hIj1gebuzg7RjMoASDaU+ygqPd7wtLtdquHVDyIkGMOhWyWuMeG9Febk9LLdORwlhGGjY80aFFGySK+4t+PPx0mpF5yixwR2UeWG8vbh0oYNDZ4c7GsAylqdR5VIawkww4E+aRYIg+PgutJ6r9UX9Dd17BL35bitWeg7hUHwgLFuFIBZF72ITpY7c0QsFFfkORgzGZuPGCZnSnpZuds108iAk2W4X6h8zH1QeeRs7/5Cdg52HHCoEwwm8g7jvw2iM= ceph-962f5a9a-b82e-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T07:29:14.514 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgOZD2X3LgS9GyqBgGlKKvA5x6UEkHQzBDs/G8WLLKOXtScyneh9bUbz8t5i9ByAG8+X1xsWFSqwfhTCdvEaq1ucHCVhN6D9fQVdQbtSY/CDpxy+hQIfL+gOfFKaeSAH8a59S1gqAX455MS7T0Ob8c+2ut9z7LXCynuKf68WqYbcSmyQ8ez8SR25/l49nxlngV/FNUONVj5N3V3jAMxGvV32/DNXfsMO5ldcH8ALEpdXK+1DCLLSBIE3KrdRMas7hIj1gebuzg7RjMoASDaU+ygqPd7wtLtdquHVDyIkGMOhWyWuMeG9Febk9LLdORwlhGGjY80aFFGySK+4t+PPx0mpF5yixwR2UeWG8vbh0oYNDZ4c7GsAylqdR5VIawkww4E+aRYIg+PgutJ6r9UX9Dd17BL35bitWeg7hUHwgLFuFIBZF72ITpY7c0QsFFfkORgzGZuPGCZnSnpZuds108iAk2W4X6h8zH1QeeRs7/5Cdg52HHCoEwwm8g7jvw2iM= ceph-962f5a9a-b82e-11ee-95b0-87774f69a715 2024-01-21T07:29:14.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 -- 172.21.15.107:0/230094717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 msgr2=0x7f96640fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 --2- 172.21.15.107:0/230094717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96640fb570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9654009fb0 tx=0x7f9654009b30).stop 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 -- 172.21.15.107:0/230094717 shutdown_connections 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 --2- 172.21.15.107:0/230094717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96640fb570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 -- 172.21.15.107:0/230094717 >> 172.21.15.107:0/230094717 conn(0x7f96640f0b70 msgr2=0x7f96640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 -- 172.21.15.107:0/230094717 shutdown_connections 2024-01-21T07:29:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.875+0000 7f966951d700 1 -- 172.21.15.107:0/230094717 wait complete. 2024-01-21T07:29:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.876+0000 7f966951d700 1 Processor -- start 2024-01-21T07:29:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.876+0000 7f966951d700 1 -- start start 2024-01-21T07:29:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.876+0000 7f966951d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.876+0000 7f966951d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9664108da0 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f9662ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f9662ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57574/0 (socket says 172.21.15.107:57574) 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f9662ffd700 1 -- 172.21.15.107:0/2615864378 learned_addr learned my addr 172.21.15.107:0/2615864378 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f9662ffd700 1 -- 172.21.15.107:0/2615864378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9654005040 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f9662ffd700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9654013510 tx=0x7f965400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9654009320 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9654013f00 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9664108fa0 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9654014dc0 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.877+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9664109430 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f965401d3d0 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f964b7fe700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9664045ad0 con 0x7f9664060200 2024-01-21T07:29:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96540410c0 con 0x7f9664060200 2024-01-21T07:29:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f96627fc700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.878+0000 7f96627fc700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 0x7f964c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f96580096a0 tx=0x7f9658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:16.882+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9654012920 con 0x7f9664060200 2024-01-21T07:29:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.038+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f96640fe150 con 0x7f9664060200 2024-01-21T07:29:17.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.042+0000 7f964b7fe700 1 -- 172.21.15.107:0/2615864378 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f965402b020 con 0x7f9664060200 2024-01-21T07:29:17.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 msgr2=0x7f964c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 0x7f964c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f96580096a0 tx=0x7f9658006b40).stop 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 msgr2=0x7f96641088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9654013510 tx=0x7f965400b040).stop 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 shutdown_connections 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f964c02ff60 0x7f964c032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 --2- 172.21.15.107:0/2615864378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9664060200 0x7f96641088b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 >> 172.21.15.107:0/2615864378 conn(0x7f96640f0b70 msgr2=0x7f96640f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 shutdown_connections 2024-01-21T07:29:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:17.044+0000 7f966951d700 1 -- 172.21.15.107:0/2615864378 wait complete. 2024-01-21T07:29:17.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:15.833450+0000 mon.smithi107 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi107.adejht(active, since 6s) 2024-01-21T07:29:17.443 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2024-01-21T07:29:17.443 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:29:17.444 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2024-01-21T07:29:17.465 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-01-21T07:29:17.465 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T07:29:17.526 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2024-01-21T07:29:17.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph orch host add smithi115 2024-01-21T07:29:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:17.043043+0000 mon.smithi107 (mon.0) 113 : audit [INF] from='client.? 172.21.15.107:0/2615864378' entity='client.admin' 2024-01-21T07:29:20.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:19.408148+0000 mon.smithi107 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:20.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:19.671625+0000 mon.smithi107 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:20.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:29:19.677845+0000 mgr.smithi107.adejht (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi107 on smithi107 2024-01-21T07:29:31.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:29.812081+0000 mgr.smithi107.adejht (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:31.812354+0000 mgr.smithi107.adejht (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:33.812663+0000 mgr.smithi107.adejht (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:35.813011+0000 mgr.smithi107.adejht (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:37.813285+0000 mgr.smithi107.adejht (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 -- 172.21.15.115:0/807126096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 msgr2=0x7f5f800fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 --2- 172.21.15.115:0/807126096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f800fb570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5f74009fb0 tx=0x7f5f74009b30).stop 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 -- 172.21.15.115:0/807126096 shutdown_connections 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 --2- 172.21.15.115:0/807126096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f800fb570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 -- 172.21.15.115:0/807126096 >> 172.21.15.115:0/807126096 conn(0x7f5f800f0b70 msgr2=0x7f5f800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:38.432 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 -- 172.21.15.115:0/807126096 shutdown_connections 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.428+0000 7f5f8638d700 1 -- 172.21.15.115:0/807126096 wait complete. 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f8638d700 1 Processor -- start 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f8638d700 1 -- start start 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f8638d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f8638d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80108db0 con 0x7f5f80060200 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f7ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f7ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47338/0 (socket says 172.21.15.115:47338) 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.429+0000 7f5f7ffff700 1 -- 172.21.15.115:0/4294140399 learned_addr learned my addr 172.21.15.115:0/4294140399 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f7ffff700 1 -- 172.21.15.115:0/4294140399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74005040 con 0x7f5f80060200 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f7ffff700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5f74013510 tx=0x7f5f7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f74009320 con 0x7f5f80060200 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f80108fb0 con 0x7f5f80060200 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f74013f00 con 0x7f5f80060200 2024-01-21T07:29:38.433 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f74014dc0 con 0x7f5f80060200 2024-01-21T07:29:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.430+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f80109440 con 0x7f5f80060200 2024-01-21T07:29:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.431+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f74014420 con 0x7f5f80060200 2024-01-21T07:29:38.434 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.431+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f80045ad0 con 0x7f5f80060200 2024-01-21T07:29:38.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.431+0000 7f5f7cff9700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 0x7f5f68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:38.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.431+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f74041110 con 0x7f5f80060200 2024-01-21T07:29:38.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.431+0000 7f5f7f7fe700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 0x7f5f68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:38.435 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.432+0000 7f5f7f7fe700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 0x7f5f68032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5f700096a0 tx=0x7f5f70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:38.437 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.436+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f740095f0 con 0x7f5f80060200 2024-01-21T07:29:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:38.604+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f800f1dc0 con 0x7f5f6802ffb0 2024-01-21T07:29:39.208 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.205+0000 7f5f7cff9700 1 -- 172.21.15.115:0/4294140399 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f5f800f1dc0 con 0x7f5f6802ffb0 2024-01-21T07:29:39.208 INFO:teuthology.orchestra.run.smithi115.stdout:Added host 'smithi115' 2024-01-21T07:29:39.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 msgr2=0x7f5f68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 0x7f5f68032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5f700096a0 tx=0x7f5f70006b40).stop 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 msgr2=0x7f5f801088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5f74013510 tx=0x7f5f7400b040).stop 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 shutdown_connections 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f6802ffb0 0x7f5f68032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 --2- 172.21.15.115:0/4294140399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f80060200 0x7f5f801088c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.207+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 >> 172.21.15.115:0/4294140399 conn(0x7f5f800f0b70 msgr2=0x7f5f800f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.208+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 shutdown_connections 2024-01-21T07:29:39.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:39.208+0000 7f5f8638d700 1 -- 172.21.15.115:0/4294140399 wait complete. 2024-01-21T07:29:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:38.606890+0000 mgr.smithi107.adejht (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:39.553 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T07:29:40.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:39.205791+0000 mon.smithi107 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:40.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:29:39.206194+0000 mgr.smithi107.adejht (mgr.14162) 12 : cephadm [INF] Added host smithi115 2024-01-21T07:29:40.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:39.813561+0000 mgr.smithi107.adejht (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 -- 172.21.15.115:0/1362537998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 msgr2=0x7f2c580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 --2- 172.21.15.115:0/1362537998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c580f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2c54009fb0 tx=0x7f2c54009b30).stop 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 -- 172.21.15.115:0/1362537998 shutdown_connections 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 --2- 172.21.15.115:0/1362537998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c580f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 -- 172.21.15.115:0/1362537998 >> 172.21.15.115:0/1362537998 conn(0x7f2c580f0b70 msgr2=0x7f2c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 -- 172.21.15.115:0/1362537998 shutdown_connections 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.673+0000 7f2c60744700 1 -- 172.21.15.115:0/1362537998 wait complete. 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c60744700 1 Processor -- start 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c60744700 1 -- start start 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c60744700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c60744700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58060220 con 0x7f2c580f5410 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c5e4e0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c5e4e0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59526/0 (socket says 172.21.15.115:59526) 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c5e4e0700 1 -- 172.21.15.115:0/3061693889 learned_addr learned my addr 172.21.15.115:0/3061693889 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.674+0000 7f2c5e4e0700 1 -- 172.21.15.115:0/3061693889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54005040 con 0x7f2c580f5410 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c5e4e0700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2c54013510 tx=0x7f2c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c54009320 con 0x7f2c580f5410 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c58060420 con 0x7f2c580f5410 2024-01-21T07:29:40.678 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c54013f00 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c54014d90 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c58109b00 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c540095f0 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.675+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c58059100 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.676+0000 7f2c4effd700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 0x7f2c44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.676+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c540410a0 con 0x7f2c580f5410 2024-01-21T07:29:40.679 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.676+0000 7f2c5dcdf700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 0x7f2c44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:40.680 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.676+0000 7f2c5dcdf700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 0x7f2c44032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2c480096a0 tx=0x7f2c48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:40.682 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.680+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c54010230 con 0x7f2c580f5410 2024-01-21T07:29:40.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.850+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c580f1d70 con 0x7f2c4402ff60 2024-01-21T07:29:40.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.852+0000 7f2c4effd700 1 -- 172.21.15.115:0/3061693889 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f2c580f1d70 con 0x7f2c4402ff60 2024-01-21T07:29:40.855 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:29:40.855 INFO:teuthology.orchestra.run.smithi115.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "smithi115", "hostname": "smithi115", "labels": [], "status": ""}] 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 msgr2=0x7f2c44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 0x7f2c44032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2c480096a0 tx=0x7f2c48006b40).stop 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 msgr2=0x7f2c58109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2c54013510 tx=0x7f2c5400b040).stop 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 shutdown_connections 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2c4402ff60 0x7f2c44032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 --2- 172.21.15.115:0/3061693889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2c580f5410 0x7f2c58109450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 >> 172.21.15.115:0/3061693889 conn(0x7f2c580f0b70 msgr2=0x7f2c580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:40.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 shutdown_connections 2024-01-21T07:29:40.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:40.854+0000 7f2c60744700 1 -- 172.21.15.115:0/3061693889 wait complete. 2024-01-21T07:29:41.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:40.853011+0000 mgr.smithi107.adejht (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T07:29:42.308 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T07:29:42.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T07:29:42.947 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:41.813912+0000 mgr.smithi107.adejht (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 -- 172.21.15.107:0/1859426474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 msgr2=0x7f37cc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 --2- 172.21.15.107:0/1859426474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc0f4be0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f37c0009fb0 tx=0x7f37c0009b30).stop 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 -- 172.21.15.107:0/1859426474 shutdown_connections 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 --2- 172.21.15.107:0/1859426474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc0f4be0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 -- 172.21.15.107:0/1859426474 >> 172.21.15.107:0/1859426474 conn(0x7f37cc0f02e0 msgr2=0x7f37cc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 -- 172.21.15.107:0/1859426474 shutdown_connections 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.437+0000 7f37d19dd700 1 -- 172.21.15.107:0/1859426474 wait complete. 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37d19dd700 1 Processor -- start 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37d19dd700 1 -- start start 2024-01-21T07:29:43.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37d19dd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37d19dd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc106bf0 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37caffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37caffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48560/0 (socket says 172.21.15.107:48560) 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37caffd700 1 -- 172.21.15.107:0/2495108401 learned_addr learned my addr 172.21.15.107:0/2495108401 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37caffd700 1 -- 172.21.15.107:0/2495108401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c0005040 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37caffd700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f37c0013510 tx=0x7f37c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.438+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c0009320 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc106df0 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37c0013f00 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c0014d90 con 0x7f37cc0f47c0 2024-01-21T07:29:43.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc107280 con 0x7f37cc0f47c0 2024-01-21T07:29:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37c00263a0 con 0x7f37cc0f47c0 2024-01-21T07:29:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37b37fe700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 0x7f37b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.439+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37c0041000 con 0x7f37cc0f47c0 2024-01-21T07:29:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.440+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc045ad0 con 0x7f37cc0f47c0 2024-01-21T07:29:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.440+0000 7f37ca7fc700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 0x7f37b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.440+0000 7f37ca7fc700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 0x7f37b4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f37bc0096a0 tx=0x7f37bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:43.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.444+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37c00095f0 con 0x7f37cc0f47c0 2024-01-21T07:29:43.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.601+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f37cc0fbf60 con 0x7f37cc0f47c0 2024-01-21T07:29:43.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.783+0000 7f37b37fe700 1 -- 172.21.15.107:0/2495108401 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f37c001db70 con 0x7f37cc0f47c0 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 msgr2=0x7f37b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 0x7f37b4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f37bc0096a0 tx=0x7f37bc006b40).stop 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 msgr2=0x7f37cc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f37c0013510 tx=0x7f37c000b040).stop 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 shutdown_connections 2024-01-21T07:29:43.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f37b402ff60 0x7f37b4032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:43.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 --2- 172.21.15.107:0/2495108401 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37cc0f47c0 0x7f37cc106700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:43.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 >> 172.21.15.107:0/2495108401 conn(0x7f37cc0f02e0 msgr2=0x7f37cc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:43.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 shutdown_connections 2024-01-21T07:29:43.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-01-21T07:29:43.785+0000 7f37d19dd700 1 -- 172.21.15.107:0/2495108401 wait complete. 2024-01-21T07:29:43.788 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-01-21T07:29:44.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:43.602689+0000 mon.smithi107 (mon.0) 117 : audit [INF] from='client.? 172.21.15.107:0/2495108401' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T07:29:44.135 INFO:tasks.cephadm:Adding mon.smithi107 on smithi107 2024-01-21T07:29:44.135 INFO:tasks.cephadm:Adding mon.smithi115 on smithi115 2024-01-21T07:29:44.135 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi107:172.21.15.107=smithi107;smithi115:172.21.15.115=smithi115' 2024-01-21T07:29:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:43.784664+0000 mon.smithi107 (mon.0) 118 : audit [INF] from='client.? 172.21.15.107:0/2495108401' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T07:29:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:43.784722+0000 mon.smithi107 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T07:29:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:43.814228+0000 mgr.smithi107.adejht (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:45.814556+0000 mgr.smithi107.adejht (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:47.814824+0000 mgr.smithi107.adejht (mgr.14162) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:49.647 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mon update... 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- 172.21.15.115:0/1082397308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 msgr2=0x7f9a1c0f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 --2- 172.21.15.115:0/1082397308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c0f7f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9a18009fb0 tx=0x7f9a18009b30).stop 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- 172.21.15.115:0/1082397308 shutdown_connections 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 --2- 172.21.15.115:0/1082397308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c0f7f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- 172.21.15.115:0/1082397308 >> 172.21.15.115:0/1082397308 conn(0x7f9a1c0f02e0 msgr2=0x7f9a1c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- 172.21.15.115:0/1082397308 shutdown_connections 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- 172.21.15.115:0/1082397308 wait complete. 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 Processor -- start 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.382+0000 7f9a23886700 1 -- start start 2024-01-21T07:29:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a23886700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a23886700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c106c30 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a21622700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a21622700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39640/0 (socket says 172.21.15.115:39640) 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a21622700 1 -- 172.21.15.115:0/2358631539 learned_addr learned my addr 172.21.15.115:0/2358631539 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.383+0000 7f9a21622700 1 -- 172.21.15.115:0/2358631539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a18005040 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a21622700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9a18013510 tx=0x7f9a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a18009320 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a18013f00 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c106e30 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a18014d90 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.384+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c1072c0 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.385+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a180095f0 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.385+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c045ad0 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.385+0000 7f9a11ffb700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 0x7f9a08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.385+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a18041000 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.385+0000 7f9a20e21700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 0x7f9a08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.386+0000 7f9a20e21700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 0x7f9a08032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.390+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a1801d7f0 con 0x7f9a1c0f7b60 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.563+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 --> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi107:172.21.15.107=smithi107;smithi115:172.21.15.115=smithi115", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a1c0f1530 con 0x7f9a0802ff60 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.569+0000 7f9a11ffb700 1 -- 172.21.15.115:0/2358631539 <== mgr.14162 v2:172.21.15.107:6800/1892560072 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9a1c0f1530 con 0x7f9a0802ff60 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 msgr2=0x7f9a08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 0x7f9a08032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).stop 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 msgr2=0x7f9a1c106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9a18013510 tx=0x7f9a1800b040).stop 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 shutdown_connections 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9a0802ff60 0x7f9a08032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 --2- 172.21.15.115:0/2358631539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9a1c0f7b60 0x7f9a1c106740 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.570+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 >> 172.21.15.115:0/2358631539 conn(0x7f9a1c0f02e0 msgr2=0x7f9a1c0f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.571+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 shutdown_connections 2024-01-21T07:29:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:49.571+0000 7f9a23886700 1 -- 172.21.15.115:0/2358631539 wait complete. 2024-01-21T07:29:50.031 DEBUG:teuthology.orchestra.run.smithi115:mon.smithi115> sudo journalctl -f -n 0 -u ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115.service 2024-01-21T07:29:50.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:29:50.034 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:29:50.061 INFO:journalctl@ceph.mon.smithi115.smithi115.stdout:-- Logs begin at Sun 2024-01-21 07:18:30 UTC. -- 2024-01-21T07:29:51.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:49.565933+0000 mgr.smithi107.adejht (mgr.14162) 19 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi107:172.21.15.107=smithi107;smithi115:172.21.15.115=smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T07:29:51.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:29:49.567389+0000 mgr.smithi107.adejht (mgr.14162) 20 : cephadm [INF] Saving service mon spec with placement smithi107:172.21.15.107=smithi107;smithi115:172.21.15.115=smithi115;count:2 2024-01-21T07:29:51.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:49.569917+0000 mon.smithi107 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:29:51.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:49.815063+0000 mgr.smithi107.adejht (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 -- 172.21.15.115:0/3329958848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 msgr2=0x7f03b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3329958848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b80f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f03a8009fb0 tx=0x7f03a8009b30).stop 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 -- 172.21.15.115:0/3329958848 shutdown_connections 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3329958848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b80f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 -- 172.21.15.115:0/3329958848 >> 172.21.15.115:0/3329958848 conn(0x7f03b80f0b70 msgr2=0x7f03b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 -- 172.21.15.115:0/3329958848 shutdown_connections 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.117+0000 7f03be1ad700 1 -- 172.21.15.115:0/3329958848 wait complete. 2024-01-21T07:29:51.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.118+0000 7f03be1ad700 1 Processor -- start 2024-01-21T07:29:51.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.118+0000 7f03be1ad700 1 -- start start 2024-01-21T07:29:51.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.118+0000 7f03be1ad700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:51.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.118+0000 7f03be1ad700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b8060220 con 0x7f03b80f5410 2024-01-21T07:29:51.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.118+0000 7f03b77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:51.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03b77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39652/0 (socket says 172.21.15.115:39652) 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03b77fe700 1 -- 172.21.15.115:0/3710918386 learned_addr learned my addr 172.21.15.115:0/3710918386 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03b77fe700 1 -- 172.21.15.115:0/3710918386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a8005040 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03b77fe700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f03a8013510 tx=0x7f03a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a8009320 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b8060420 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03a8013f00 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a8014d90 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.119+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b8109aa0 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.120+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03a80095f0 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.120+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b8059100 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.120+0000 7f039ffff700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 0x7f03a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.120+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03a8041080 con 0x7f03b80f5410 2024-01-21T07:29:51.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.120+0000 7f03b6ffd700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 0x7f03a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:51.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.121+0000 7f03b6ffd700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 0x7f03a0032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:51.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.125+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03a8014420 con 0x7f03b80f5410 2024-01-21T07:29:51.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.331+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03b80fae60 con 0x7f03b80f5410 2024-01-21T07:29:51.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.332+0000 7f039ffff700 1 -- 172.21.15.115:0/3710918386 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f03a8010230 con 0x7f03b80f5410 2024-01-21T07:29:51.334 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:29:51.334 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 msgr2=0x7f03a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 0x7f03a0032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).stop 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 msgr2=0x7f03b81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f03a8013510 tx=0x7f03a800b040).stop 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 shutdown_connections 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f03a002ff60 0x7f03a0032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 --2- 172.21.15.115:0/3710918386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b80f5410 0x7f03b81093f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.333+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 >> 172.21.15.115:0/3710918386 conn(0x7f03b80f0b70 msgr2=0x7f03b80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.334+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 shutdown_connections 2024-01-21T07:29:51.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:51.334+0000 7f03be1ad700 1 -- 172.21.15.115:0/3710918386 wait complete. 2024-01-21T07:29:51.337 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:29:52.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:51.333026+0000 mon.smithi107 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.115:0/3710918386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:29:52.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:29:52.715 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:29:53.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:51.815435+0000 mgr.smithi107.adejht (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:53.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- 172.21.15.115:0/246551691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 msgr2=0x7f1f840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:53.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 --2- 172.21.15.115:0/246551691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f840f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1f78009fb0 tx=0x7f1f78009b30).stop 2024-01-21T07:29:53.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- 172.21.15.115:0/246551691 shutdown_connections 2024-01-21T07:29:53.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 --2- 172.21.15.115:0/246551691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f840f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:53.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- 172.21.15.115:0/246551691 >> 172.21.15.115:0/246551691 conn(0x7f1f840f09a0 msgr2=0x7f1f840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- 172.21.15.115:0/246551691 shutdown_connections 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- 172.21.15.115:0/246551691 wait complete. 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 Processor -- start 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.885+0000 7f1f8359e700 1 -- start start 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8359e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8359e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84060220 con 0x7f1f840f51a0 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39662/0 (socket says 172.21.15.115:39662) 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8259c700 1 -- 172.21.15.115:0/2347019436 learned_addr learned my addr 172.21.15.115:0/2347019436 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8259c700 1 -- 172.21.15.115:0/2347019436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78005040 con 0x7f1f840f51a0 2024-01-21T07:29:53.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8259c700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1f78009830 tx=0x7f1f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:53.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f78009320 con 0x7f1f840f51a0 2024-01-21T07:29:53.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f780136c0 con 0x7f1f840f51a0 2024-01-21T07:29:53.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f84060420 con 0x7f1f840f51a0 2024-01-21T07:29:53.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f7801ee90 con 0x7f1f840f51a0 2024-01-21T07:29:53.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.886+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f84109880 con 0x7f1f840f51a0 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.887+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f7801e420 con 0x7f1f840f51a0 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.887+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f84059100 con 0x7f1f840f51a0 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.887+0000 7f1f72ffd700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 0x7f1f6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.888+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f78040f40 con 0x7f1f840f51a0 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.888+0000 7f1f81d9b700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 0x7f1f6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.888+0000 7f1f81d9b700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 0x7f1f6c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1f740096a0 tx=0x7f1f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:53.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:53.892+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f7806d2b0 con 0x7f1f840f51a0 2024-01-21T07:29:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.096+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f84045ad0 con 0x7f1f840f51a0 2024-01-21T07:29:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.096+0000 7f1f72ffd700 1 -- 172.21.15.115:0/2347019436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f780095f0 con 0x7f1f840f51a0 2024-01-21T07:29:54.099 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:29:54.099 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 msgr2=0x7f1f6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 0x7f1f6c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1f740096a0 tx=0x7f1f74006b40).stop 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 msgr2=0x7f1f841091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1f78009830 tx=0x7f1f7800b040).stop 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 shutdown_connections 2024-01-21T07:29:54.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1f6c02ff10 0x7f1f6c0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:54.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 --2- 172.21.15.115:0/2347019436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f840f51a0 0x7f1f841091d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:54.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 >> 172.21.15.115:0/2347019436 conn(0x7f1f840f09a0 msgr2=0x7f1f840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:54.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 shutdown_connections 2024-01-21T07:29:54.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:54.098+0000 7f1f8359e700 1 -- 172.21.15.115:0/2347019436 wait complete. 2024-01-21T07:29:54.101 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:29:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:53.815796+0000 mgr.smithi107.adejht (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:54.097793+0000 mon.smithi107 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.115:0/2347019436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:29:55.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:29:55.486 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:29:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:55.816114+0000 mgr.smithi107.adejht (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:56.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 -- 172.21.15.115:0/4230988767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 msgr2=0x7fc2740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:56.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 --2- 172.21.15.115:0/4230988767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc2740f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc264009fb0 tx=0x7fc264009b30).stop 2024-01-21T07:29:56.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 -- 172.21.15.115:0/4230988767 shutdown_connections 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 --2- 172.21.15.115:0/4230988767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc2740f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 -- 172.21.15.115:0/4230988767 >> 172.21.15.115:0/4230988767 conn(0x7fc2740f0b70 msgr2=0x7fc2740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 -- 172.21.15.115:0/4230988767 shutdown_connections 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.655+0000 7fc27c115700 1 -- 172.21.15.115:0/4230988767 wait complete. 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc27c115700 1 Processor -- start 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc27c115700 1 -- start start 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc27c115700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc27c115700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274105b20 con 0x7fc2740f5410 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc279eb1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc279eb1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39668/0 (socket says 172.21.15.115:39668) 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc279eb1700 1 -- 172.21.15.115:0/3271268871 learned_addr learned my addr 172.21.15.115:0/3271268871 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.656+0000 7fc279eb1700 1 -- 172.21.15.115:0/3271268871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc264005040 con 0x7fc2740f5410 2024-01-21T07:29:56.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc279eb1700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc264013510 tx=0x7fc26400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:56.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc264009320 con 0x7fc2740f5410 2024-01-21T07:29:56.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274105d20 con 0x7fc2740f5410 2024-01-21T07:29:56.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc264013f00 con 0x7fc2740f5410 2024-01-21T07:29:56.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc264014d90 con 0x7fc2740f5410 2024-01-21T07:29:56.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.657+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741061b0 con 0x7fc2740f5410 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2640095f0 con 0x7fc2740f5410 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc274045ad0 con 0x7fc2740f5410 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc26a7fc700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2640410c0 con 0x7fc2740f5410 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc2796b0700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:56.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.658+0000 7fc2796b0700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 0x7fc260032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc2700096a0 tx=0x7fc270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:56.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.663+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc264014420 con 0x7fc2740f5410 2024-01-21T07:29:56.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.871+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2740fae60 con 0x7fc2740f5410 2024-01-21T07:29:56.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.872+0000 7fc26a7fc700 1 -- 172.21.15.115:0/3271268871 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc26401d7f0 con 0x7fc2740f5410 2024-01-21T07:29:56.874 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:29:56.874 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 msgr2=0x7fc260032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 0x7fc260032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc2700096a0 tx=0x7fc270006b40).stop 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 msgr2=0x7fc274105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc264013510 tx=0x7fc26400b040).stop 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 shutdown_connections 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 --2- 172.21.15.115:0/3271268871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2740f5410 0x7fc274105630 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 >> 172.21.15.115:0/3271268871 conn(0x7fc2740f0b70 msgr2=0x7fc2740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 shutdown_connections 2024-01-21T07:29:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:56.873+0000 7fc27c115700 1 -- 172.21.15.115:0/3271268871 wait complete. 2024-01-21T07:29:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:29:58.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:56.873316+0000 mon.smithi107 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.115:0/3271268871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:29:58.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:29:58.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:29:59.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:57.816478+0000 mgr.smithi107.adejht (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.314+0000 7f42b0466700 1 -- 172.21.15.115:0/4076369022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 msgr2=0x7f42a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.314+0000 7f42b0466700 1 --2- 172.21.15.115:0/4076369022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a80f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4298009fb0 tx=0x7f4298009b30).stop 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.314+0000 7f42b0466700 1 -- 172.21.15.115:0/4076369022 shutdown_connections 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.314+0000 7f42b0466700 1 --2- 172.21.15.115:0/4076369022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a80f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.314+0000 7f42b0466700 1 -- 172.21.15.115:0/4076369022 >> 172.21.15.115:0/4076369022 conn(0x7f42a80f0b70 msgr2=0x7f42a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 -- 172.21.15.115:0/4076369022 shutdown_connections 2024-01-21T07:29:59.318 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 -- 172.21.15.115:0/4076369022 wait complete. 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 Processor -- start 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 -- start start 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42b0466700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8106bf0 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42ae202700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42ae202700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39686/0 (socket says 172.21.15.115:39686) 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.315+0000 7f42ae202700 1 -- 172.21.15.115:0/4219644811 learned_addr learned my addr 172.21.15.115:0/4219644811 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f42ae202700 1 -- 172.21.15.115:0/4219644811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298005040 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f42ae202700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4298013510 tx=0x7f429800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4298009320 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a8106df0 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4298013f00 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4298014d90 con 0x7f42a80f5410 2024-01-21T07:29:59.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.316+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a8107280 con 0x7f42a80f5410 2024-01-21T07:29:59.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.317+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4298014420 con 0x7f42a80f5410 2024-01-21T07:29:59.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.317+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a8045ad0 con 0x7f42a80f5410 2024-01-21T07:29:59.321 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.317+0000 7f429effd700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 0x7f4294032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:29:59.321 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.317+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4298040fd0 con 0x7f42a80f5410 2024-01-21T07:29:59.321 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.317+0000 7f42ada01700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 0x7f4294032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:29:59.321 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.318+0000 7f42ada01700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 0x7f4294032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f42a40096a0 tx=0x7f42a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:29:59.324 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.322+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f429806d2b0 con 0x7f42a80f5410 2024-01-21T07:29:59.534 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.532+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42a80fae60 con 0x7f42a80f5410 2024-01-21T07:29:59.535 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.533+0000 7f429effd700 1 -- 172.21.15.115:0/4219644811 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42980095f0 con 0x7f42a80f5410 2024-01-21T07:29:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:29:59.535 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 msgr2=0x7f4294032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 0x7f4294032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f42a40096a0 tx=0x7f42a4006b40).stop 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 msgr2=0x7f42a8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4298013510 tx=0x7f429800b040).stop 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 shutdown_connections 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f429402ffb0 0x7f4294032470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 --2- 172.21.15.115:0/4219644811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42a80f5410 0x7f42a8106700 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.534+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 >> 172.21.15.115:0/4219644811 conn(0x7f42a80f0b70 msgr2=0x7f42a80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.535+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 shutdown_connections 2024-01-21T07:29:59.537 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:29:59.535+0000 7f42b0466700 1 -- 172.21.15.115:0/4219644811 wait complete. 2024-01-21T07:29:59.538 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:29:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:29:59.534154+0000 mon.smithi107 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.115:0/4219644811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:00.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:00.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:29:59.816793+0000 mgr.smithi107.adejht (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:02.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/200663181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 msgr2=0x7f0d840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:02.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/200663181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d840f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0d74009fb0 tx=0x7f0d74009b30).stop 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/200663181 shutdown_connections 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/200663181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d840f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/200663181 >> 172.21.15.115:0/200663181 conn(0x7f0d840f1c70 msgr2=0x7f0d840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/200663181 shutdown_connections 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.048+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/200663181 wait complete. 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d8a1bd700 1 Processor -- start 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d8a1bd700 1 -- start start 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d8a1bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d8a1bd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84106bc0 con 0x7f0d840f6510 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:02.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d837fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59574/0 (socket says 172.21.15.115:59574) 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.049+0000 7f0d837fe700 1 -- 172.21.15.115:0/1523244688 learned_addr learned my addr 172.21.15.115:0/1523244688 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d837fe700 1 -- 172.21.15.115:0/1523244688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d74005040 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d837fe700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0d74013510 tx=0x7f0d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d74009320 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d84106dc0 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d74013f00 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d74014d90 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.050+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d84107250 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d740095f0 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d84045ad0 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d6bfff700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 0x7f0d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d74041080 con 0x7f0d840f6510 2024-01-21T07:30:02.055 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d82ffd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 0x7f0d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:02.056 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.051+0000 7f0d82ffd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 0x7f0d6c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0d780096a0 tx=0x7f0d78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:02.057 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.056+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d74014420 con 0x7f0d840f6510 2024-01-21T07:30:02.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.261+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d841080a0 con 0x7f0d840f6510 2024-01-21T07:30:02.264 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.262+0000 7f0d6bfff700 1 -- 172.21.15.115:0/1523244688 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d74010230 con 0x7f0d840f6510 2024-01-21T07:30:02.265 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:02.265 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:02.266 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 msgr2=0x7f0d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:02.266 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 0x7f0d6c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0d780096a0 tx=0x7f0d78006b40).stop 2024-01-21T07:30:02.266 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 msgr2=0x7f0d841066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:02.266 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0d74013510 tx=0x7f0d7400b040).stop 2024-01-21T07:30:02.266 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 shutdown_connections 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0d6c02ff60 0x7f0d6c032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 --2- 172.21.15.115:0/1523244688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d840f6510 0x7f0d841066d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 >> 172.21.15.115:0/1523244688 conn(0x7f0d840f1c70 msgr2=0x7f0d840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 shutdown_connections 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:02.264+0000 7f0d8a1bd700 1 -- 172.21.15.115:0/1523244688 wait complete. 2024-01-21T07:30:02.267 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:01.817143+0000 mgr.smithi107.adejht (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:02.263605+0000 mon.smithi107 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.115:0/1523244688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:03.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:03.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:03.817491+0000 mgr.smithi107.adejht (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:04.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 -- 172.21.15.115:0/4015950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 msgr2=0x7f9d880f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:04.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/4015950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d880f6e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9d7c009fb0 tx=0x7f9d7c009b30).stop 2024-01-21T07:30:04.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 -- 172.21.15.115:0/4015950638 shutdown_connections 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/4015950638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d880f6e00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 -- 172.21.15.115:0/4015950638 >> 172.21.15.115:0/4015950638 conn(0x7f9d880f02e0 msgr2=0x7f9d880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 -- 172.21.15.115:0/4015950638 shutdown_connections 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.796+0000 7f9d8ca07700 1 -- 172.21.15.115:0/4015950638 wait complete. 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8ca07700 1 Processor -- start 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8ca07700 1 -- start start 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8ca07700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8ca07700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88105ae0 con 0x7f9d880f69e0 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59600/0 (socket says 172.21.15.115:59600) 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.797+0000 7f9d8659c700 1 -- 172.21.15.115:0/3699938160 learned_addr learned my addr 172.21.15.115:0/3699938160 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d8659c700 1 -- 172.21.15.115:0/3699938160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c005040 con 0x7f9d880f69e0 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d8659c700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9d7c013510 tx=0x7f9d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:04.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d7c009320 con 0x7f9d880f69e0 2024-01-21T07:30:04.802 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88105ce0 con 0x7f9d880f69e0 2024-01-21T07:30:04.802 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d7c013f00 con 0x7f9d880f69e0 2024-01-21T07:30:04.802 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d7c014d90 con 0x7f9d880f69e0 2024-01-21T07:30:04.802 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.798+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d88106170 con 0x7f9d880f69e0 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.799+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d7c0095f0 con 0x7f9d880f69e0 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.799+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d88045ad0 con 0x7f9d880f69e0 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.799+0000 7f9d76ffd700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.799+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d7c0410c0 con 0x7f9d880f69e0 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.799+0000 7f9d85d9b700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:04.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.800+0000 7f9d85d9b700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 0x7f9d70032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9d780096a0 tx=0x7f9d78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:04.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:04.803+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d7c014420 con 0x7f9d880f69e0 2024-01-21T07:30:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.011+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d88107020 con 0x7f9d880f69e0 2024-01-21T07:30:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.012+0000 7f9d76ffd700 1 -- 172.21.15.115:0/3699938160 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d7c010230 con 0x7f9d880f69e0 2024-01-21T07:30:05.014 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:05.014 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.013+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 msgr2=0x7f9d70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.013+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 0x7f9d70032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9d780096a0 tx=0x7f9d78006b40).stop 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.013+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 msgr2=0x7f9d881055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.013+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9d7c013510 tx=0x7f9d7c00b040).stop 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 shutdown_connections 2024-01-21T07:30:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d7002ff60 0x7f9d70032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 --2- 172.21.15.115:0/3699938160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d880f69e0 0x7f9d881055f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 >> 172.21.15.115:0/3699938160 conn(0x7f9d880f02e0 msgr2=0x7f9d880f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 shutdown_connections 2024-01-21T07:30:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:05.014+0000 7f9d8ca07700 1 -- 172.21.15.115:0/3699938160 wait complete. 2024-01-21T07:30:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:05.013289+0000 mon.smithi107 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.115:0/3699938160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:06.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:06.418 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:06.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:05.817861+0000 mgr.smithi107.adejht (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1270111614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 msgr2=0x7fa0b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1270111614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b40f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fa0a8009fb0 tx=0x7fa0a8009b30).stop 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1270111614 shutdown_connections 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1270111614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b40f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1270111614 >> 172.21.15.115:0/1270111614 conn(0x7fa0b40f1c70 msgr2=0x7fa0b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1270111614 shutdown_connections 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1270111614 wait complete. 2024-01-21T07:30:07.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.627+0000 7fa0b9bc5700 1 Processor -- start 2024-01-21T07:30:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b9bc5700 1 -- start start 2024-01-21T07:30:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b9bc5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:07.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b9bc5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4106c20 con 0x7fa0b40f6510 2024-01-21T07:30:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59614/0 (socket says 172.21.15.115:59614) 2024-01-21T07:30:07.631 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b37fe700 1 -- 172.21.15.115:0/1005492771 learned_addr learned my addr 172.21.15.115:0/1005492771 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.628+0000 7fa0b37fe700 1 -- 172.21.15.115:0/1005492771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8005040 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa0b37fe700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa0a8013510 tx=0x7fa0a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a8009320 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b4106e20 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0a8013f00 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a8014d90 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.629+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41072b0 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0a80263a0 con 0x7fa0b40f6510 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa09bfff700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0a80410e0 con 0x7fa0b40f6510 2024-01-21T07:30:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b4045ad0 con 0x7fa0b40f6510 2024-01-21T07:30:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa0b2ffd700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.630+0000 7fa0b2ffd700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 0x7fa09c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:07.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.635+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0a8012920 con 0x7fa0b40f6510 2024-01-21T07:30:07.840 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.839+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0b40fc050 con 0x7fa0b40f6510 2024-01-21T07:30:07.841 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.840+0000 7fa09bfff700 1 -- 172.21.15.115:0/1005492771 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa0a802b020 con 0x7fa0b40f6510 2024-01-21T07:30:07.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:07.841 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 msgr2=0x7fa09c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 0x7fa09c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).stop 2024-01-21T07:30:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 msgr2=0x7fa0b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa0a8013510 tx=0x7fa0a800b040).stop 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 shutdown_connections 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 --2- 172.21.15.115:0/1005492771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0b40f6510 0x7fa0b4106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 >> 172.21.15.115:0/1005492771 conn(0x7fa0b40f1c70 msgr2=0x7fa0b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 shutdown_connections 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:07.842+0000 7fa0b9bc5700 1 -- 172.21.15.115:0/1005492771 wait complete. 2024-01-21T07:30:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:07.818132+0000 mgr.smithi107.adejht (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:07.840377+0000 mon.smithi107 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.115:0/1005492771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:09.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:09.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:10.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:09.818479+0000 mgr.smithi107.adejht (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:09.945090+0000 mon.smithi107 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:30:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:09.946950+0000 mon.smithi107 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.066+0000 7f5b8fee0700 1 -- 172.21.15.115:0/2297621785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 msgr2=0x7f5b880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.066+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/2297621785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b880f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f5b78009fb0 tx=0x7f5b78009b30).stop 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.066+0000 7f5b8fee0700 1 -- 172.21.15.115:0/2297621785 shutdown_connections 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.066+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/2297621785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b880f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.066+0000 7f5b8fee0700 1 -- 172.21.15.115:0/2297621785 >> 172.21.15.115:0/2297621785 conn(0x7f5b880f0b70 msgr2=0x7f5b880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 -- 172.21.15.115:0/2297621785 shutdown_connections 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 -- 172.21.15.115:0/2297621785 wait complete. 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 Processor -- start 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 -- start start 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8fee0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88060220 con 0x7f5b880f5410 2024-01-21T07:30:11.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8dc7c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:11.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8dc7c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45686/0 (socket says 172.21.15.115:45686) 2024-01-21T07:30:11.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.067+0000 7f5b8dc7c700 1 -- 172.21.15.115:0/1189116894 learned_addr learned my addr 172.21.15.115:0/1189116894 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:11.335 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:11.335 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:11.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b8dc7c700 1 -- 172.21.15.115:0/1189116894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78005040 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b8dc7c700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5b78013510 tx=0x7f5b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b78009320 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b88060420 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b78013f00 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b78014d90 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.068+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b88109b00 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.069+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b78014420 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.069+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b88059100 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.069+0000 7f5b7e7fc700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.069+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b78041000 con 0x7f5b880f5410 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.069+0000 7f5b8d47b700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:11.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.070+0000 7f5b8d47b700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 0x7f5b74032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5b840096a0 tx=0x7f5b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.074+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b78026bf0 con 0x7f5b880f5410 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.278+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b88045ad0 con 0x7f5b880f5410 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.279+0000 7f5b7e7fc700 1 -- 172.21.15.115:0/1189116894 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b7801db70 con 0x7f5b880f5410 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 msgr2=0x7f5b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 0x7f5b74032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5b840096a0 tx=0x7f5b84006b40).stop 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 msgr2=0x7f5b88109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5b78013510 tx=0x7f5b7800b040).stop 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 shutdown_connections 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 --2- 172.21.15.115:0/1189116894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b880f5410 0x7f5b88109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 >> 172.21.15.115:0/1189116894 conn(0x7f5b880f0b70 msgr2=0x7f5b880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 shutdown_connections 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:11.281+0000 7f5b8fee0700 1 -- 172.21.15.115:0/1189116894 wait complete. 2024-01-21T07:30:11.337 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:12.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:11.279455+0000 mon.smithi107 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.115:0/1189116894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:12.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:11.818847+0000 mgr.smithi107.adejht (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:13.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:13.642 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:14.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:13.819110+0000 mgr.smithi107.adejht (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:14.740 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 -- 172.21.15.115:0/714955742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 msgr2=0x7fc6140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:14.740 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 --2- 172.21.15.115:0/714955742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6140f55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc608009fb0 tx=0x7fc608009b30).stop 2024-01-21T07:30:14.740 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 -- 172.21.15.115:0/714955742 shutdown_connections 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 --2- 172.21.15.115:0/714955742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6140f55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 -- 172.21.15.115:0/714955742 >> 172.21.15.115:0/714955742 conn(0x7fc6140f09a0 msgr2=0x7fc6140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 -- 172.21.15.115:0/714955742 shutdown_connections 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.738+0000 7fc619a03700 1 -- 172.21.15.115:0/714955742 wait complete. 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc619a03700 1 Processor -- start 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc619a03700 1 -- start start 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc619a03700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc619a03700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc614060220 con 0x7fc6140f51a0 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc618a01700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc618a01700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45694/0 (socket says 172.21.15.115:45694) 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc618a01700 1 -- 172.21.15.115:0/500462117 learned_addr learned my addr 172.21.15.115:0/500462117 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:14.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.739+0000 7fc618a01700 1 -- 172.21.15.115:0/500462117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc608005040 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc618a01700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc608013510 tx=0x7fc60800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc608009320 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc608013f00 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc614060420 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc608014dc0 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.740+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc614109880 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc608014420 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc614059100 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc6117fa700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 0x7fc604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc608041030 con 0x7fc6140f51a0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc613fff700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 0x7fc604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:14.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.741+0000 7fc613fff700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 0x7fc604032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc6000096a0 tx=0x7fc600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:14.747 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.746+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc608026bf0 con 0x7fc6140f51a0 2024-01-21T07:30:14.955 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.953+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc614045ad0 con 0x7fc6140f51a0 2024-01-21T07:30:14.955 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.954+0000 7fc6117fa700 1 -- 172.21.15.115:0/500462117 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc60801cb10 con 0x7fc6140f51a0 2024-01-21T07:30:14.955 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:14.956 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 msgr2=0x7fc604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 0x7fc604032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fc6000096a0 tx=0x7fc600006b40).stop 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 msgr2=0x7fc6141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc608013510 tx=0x7fc60800b040).stop 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 shutdown_connections 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc60402ff60 0x7fc604032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 --2- 172.21.15.115:0/500462117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6140f51a0 0x7fc6141091d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 >> 172.21.15.115:0/500462117 conn(0x7fc6140f09a0 msgr2=0x7fc6140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 shutdown_connections 2024-01-21T07:30:14.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:14.956+0000 7fc619a03700 1 -- 172.21.15.115:0/500462117 wait complete. 2024-01-21T07:30:14.958 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:15.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:14.954516+0000 mon.smithi107 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.115:0/500462117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:15.819350+0000 mgr.smithi107.adejht (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:16.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:16.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:17.499 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 -- 172.21.15.115:0/3870599974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 msgr2=0x7f67cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:17.499 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 --2- 172.21.15.115:0/3870599974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc0f55c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f67bc009fb0 tx=0x7f67bc009b30).stop 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 -- 172.21.15.115:0/3870599974 shutdown_connections 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 --2- 172.21.15.115:0/3870599974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc0f55c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 -- 172.21.15.115:0/3870599974 >> 172.21.15.115:0/3870599974 conn(0x7f67cc0f09a0 msgr2=0x7f67cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 -- 172.21.15.115:0/3870599974 shutdown_connections 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.497+0000 7f67cbfff700 1 -- 172.21.15.115:0/3870599974 wait complete. 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67cbfff700 1 Processor -- start 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67cbfff700 1 -- start start 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67cbfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67cbfff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc060220 con 0x7f67cc0f51a0 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67caffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67caffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45720/0 (socket says 172.21.15.115:45720) 2024-01-21T07:30:17.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.498+0000 7f67caffd700 1 -- 172.21.15.115:0/901805576 learned_addr learned my addr 172.21.15.115:0/901805576 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67caffd700 1 -- 172.21.15.115:0/901805576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc005040 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67caffd700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67bc013510 tx=0x7f67bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc009320 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc060420 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67bc013f00 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc014dc0 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.499+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc109880 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67bc014420 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc059100 con 0x7f67cc0f51a0 2024-01-21T07:30:17.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67b37fe700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:17.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67bc0410f0 con 0x7f67cc0f51a0 2024-01-21T07:30:17.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67ca7fc700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:17.503 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.500+0000 7f67ca7fc700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 0x7f67b4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f67c00096a0 tx=0x7f67c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:17.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.505+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67bc0095f0 con 0x7f67cc0f51a0 2024-01-21T07:30:17.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.712+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67cc045ad0 con 0x7f67cc0f51a0 2024-01-21T07:30:17.714 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.713+0000 7f67b37fe700 1 -- 172.21.15.115:0/901805576 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67bc01cb10 con 0x7f67cc0f51a0 2024-01-21T07:30:17.714 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:17.714 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:17.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 msgr2=0x7f67b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:17.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 0x7f67b4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f67c00096a0 tx=0x7f67c0006b40).stop 2024-01-21T07:30:17.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 msgr2=0x7f67cc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:17.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67bc013510 tx=0x7f67bc00b040).stop 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 shutdown_connections 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 --2- 172.21.15.115:0/901805576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f67cc0f51a0 0x7f67cc1091d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 >> 172.21.15.115:0/901805576 conn(0x7f67cc0f09a0 msgr2=0x7f67cc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 shutdown_connections 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:17.714+0000 7f67cbfff700 1 -- 172.21.15.115:0/901805576 wait complete. 2024-01-21T07:30:17.717 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:18.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:17.712996+0000 mon.smithi107 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.115:0/901805576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:19.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:17.819672+0000 mgr.smithi107.adejht (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:19.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:19.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:20.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:19.819963+0000 mgr.smithi107.adejht (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 -- 172.21.15.115:0/3373406424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 msgr2=0x7fd0800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 --2- 172.21.15.115:0/3373406424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd0800f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd074009fb0 tx=0x7fd074009b30).stop 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 -- 172.21.15.115:0/3373406424 shutdown_connections 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 --2- 172.21.15.115:0/3373406424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd0800f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 -- 172.21.15.115:0/3373406424 >> 172.21.15.115:0/3373406424 conn(0x7fd0800f0b70 msgr2=0x7fd0800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 -- 172.21.15.115:0/3373406424 shutdown_connections 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 -- 172.21.15.115:0/3373406424 wait complete. 2024-01-21T07:30:20.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.345+0000 7fd087153700 1 Processor -- start 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd087153700 1 -- start start 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd087153700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd087153700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd080060220 con 0x7fd0800f5410 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd084eef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd084eef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51816/0 (socket says 172.21.15.115:51816) 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd084eef700 1 -- 172.21.15.115:0/2985226094 learned_addr learned my addr 172.21.15.115:0/2985226094 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd084eef700 1 -- 172.21.15.115:0/2985226094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074005040 con 0x7fd0800f5410 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.346+0000 7fd084eef700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd074014000 tx=0x7fd07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0740095f0 con 0x7fd0800f5410 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd080060420 con 0x7fd0800f5410 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd074007e70 con 0x7fd0800f5410 2024-01-21T07:30:20.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd074014ea0 con 0x7fd0800f5410 2024-01-21T07:30:20.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd080109b00 con 0x7fd0800f5410 2024-01-21T07:30:20.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.347+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd074025070 con 0x7fd0800f5410 2024-01-21T07:30:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.348+0000 7fd07d7fa700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 0x7fd070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.348+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd080059100 con 0x7fd0800f5410 2024-01-21T07:30:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.348+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd074040930 con 0x7fd0800f5410 2024-01-21T07:30:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.348+0000 7fd07ffff700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 0x7fd070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:20.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.348+0000 7fd07ffff700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 0x7fd070032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:20.354 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.353+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd07400f4e0 con 0x7fd0800f5410 2024-01-21T07:30:20.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.560+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd080045ad0 con 0x7fd0800f5410 2024-01-21T07:30:20.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.561+0000 7fd07d7fa700 1 -- 172.21.15.115:0/2985226094 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd07400f300 con 0x7fd0800f5410 2024-01-21T07:30:20.563 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:20.563 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:20.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 msgr2=0x7fd070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 0x7fd070032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).stop 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 msgr2=0x7fd080109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd074014000 tx=0x7fd07400b040).stop 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 shutdown_connections 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd07002ff60 0x7fd070032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 --2- 172.21.15.115:0/2985226094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0800f5410 0x7fd080109450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 >> 172.21.15.115:0/2985226094 conn(0x7fd0800f0b70 msgr2=0x7fd0800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 shutdown_connections 2024-01-21T07:30:20.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:20.562+0000 7fd087153700 1 -- 172.21.15.115:0/2985226094 wait complete. 2024-01-21T07:30:20.566 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:21.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:20.562258+0000 mon.smithi107 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.115:0/2985226094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:21.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:21.949 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:23.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:21.820240+0000 mgr.smithi107.adejht (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:23.346 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.341+0000 7f72f5145700 1 -- 172.21.15.115:0/2847472203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 msgr2=0x7f72f00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:23.346 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.341+0000 7f72f5145700 1 --2- 172.21.15.115:0/2847472203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f00f9190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f72e0009fb0 tx=0x7f72e0009b30).stop 2024-01-21T07:30:23.346 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.341+0000 7f72f5145700 1 -- 172.21.15.115:0/2847472203 shutdown_connections 2024-01-21T07:30:23.346 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.341+0000 7f72f5145700 1 --2- 172.21.15.115:0/2847472203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f00f9190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:23.346 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.341+0000 7f72f5145700 1 -- 172.21.15.115:0/2847472203 >> 172.21.15.115:0/2847472203 conn(0x7f72f00f0b70 msgr2=0x7f72f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 -- 172.21.15.115:0/2847472203 shutdown_connections 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 -- 172.21.15.115:0/2847472203 wait complete. 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 Processor -- start 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 -- start start 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72f5145700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f0106bd0 con 0x7f72f00601e0 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.342+0000 7f72eed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72eed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51834/0 (socket says 172.21.15.115:51834) 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72eed9d700 1 -- 172.21.15.115:0/214702851 learned_addr learned my addr 172.21.15.115:0/214702851 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72eed9d700 1 -- 172.21.15.115:0/214702851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e0005040 con 0x7f72f00601e0 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72eed9d700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f72e0004dd0 tx=0x7f72e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:23.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e0018070 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f0106dd0 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72e0013410 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e001c3c0 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.343+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f0107260 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.344+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72e001c630 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.344+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f0045ad0 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.344+0000 7f72d77fe700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 0x7f72d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.344+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72e0041680 con 0x7f72f00601e0 2024-01-21T07:30:23.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.344+0000 7f72ee59c700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 0x7f72d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:23.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.345+0000 7f72ee59c700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 0x7f72d8032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f72e40096a0 tx=0x7f72e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:23.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.349+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72e001ca30 con 0x7f72f00601e0 2024-01-21T07:30:23.555 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.553+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72f0108110 con 0x7f72f00601e0 2024-01-21T07:30:23.556 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.554+0000 7f72d77fe700 1 -- 172.21.15.115:0/214702851 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f72e001b370 con 0x7f72f00601e0 2024-01-21T07:30:23.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:23.556 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.555+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 msgr2=0x7f72d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.555+0000 7f72f5145700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 0x7f72d8032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f72e40096a0 tx=0x7f72e4006b40).stop 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.555+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 msgr2=0x7f72f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.555+0000 7f72f5145700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f72e0004dd0 tx=0x7f72e0009b30).stop 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.555+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 shutdown_connections 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.556+0000 7f72f5145700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f72d802ff60 0x7f72d8032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.556+0000 7f72f5145700 1 --2- 172.21.15.115:0/214702851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f00601e0 0x7f72f01066e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.556+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 >> 172.21.15.115:0/214702851 conn(0x7f72f00f0b70 msgr2=0x7f72f00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.556+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 shutdown_connections 2024-01-21T07:30:23.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:23.556+0000 7f72f5145700 1 -- 172.21.15.115:0/214702851 wait complete. 2024-01-21T07:30:23.559 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:24.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:23.555080+0000 mon.smithi107 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.115:0/214702851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:24.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:24.928 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:25.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:23.820586+0000 mgr.smithi107.adejht (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 -- 172.21.15.115:0/606791549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 msgr2=0x7f66b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 --2- 172.21.15.115:0/606791549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b00f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f66a4009fb0 tx=0x7f66a4009b30).stop 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 -- 172.21.15.115:0/606791549 shutdown_connections 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 --2- 172.21.15.115:0/606791549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b00f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 -- 172.21.15.115:0/606791549 >> 172.21.15.115:0/606791549 conn(0x7f66b00f1c70 msgr2=0x7f66b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:26.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 -- 172.21.15.115:0/606791549 shutdown_connections 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.081+0000 7f66b8580700 1 -- 172.21.15.115:0/606791549 wait complete. 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b8580700 1 Processor -- start 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b8580700 1 -- start start 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b8580700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b8580700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0106c20 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b631c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b631c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51848/0 (socket says 172.21.15.115:51848) 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b631c700 1 -- 172.21.15.115:0/759643048 learned_addr learned my addr 172.21.15.115:0/759643048 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.082+0000 7f66b631c700 1 -- 172.21.15.115:0/759643048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4005040 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66b631c700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f66a4014000 tx=0x7f66a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a40095f0 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0106e20 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a4007e70 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a4014ea0 con 0x7f66b00f6510 2024-01-21T07:30:26.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b01072b0 con 0x7f66b00f6510 2024-01-21T07:30:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66a4014690 con 0x7f66b00f6510 2024-01-21T07:30:26.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.083+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b0045ad0 con 0x7f66b00f6510 2024-01-21T07:30:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.084+0000 7f66a2ffd700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 0x7f669c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.084+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66a4040930 con 0x7f66b00f6510 2024-01-21T07:30:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.084+0000 7f66b5b1b700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 0x7f669c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.084+0000 7f66b5b1b700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 0x7f669c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f66ac0096a0 tx=0x7f66ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:26.090 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.088+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66a400f300 con 0x7f66b00f6510 2024-01-21T07:30:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.296+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66b00fbf60 con 0x7f66b00f6510 2024-01-21T07:30:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.297+0000 7f66a2ffd700 1 -- 172.21.15.115:0/759643048 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66a400f300 con 0x7f66b00f6510 2024-01-21T07:30:26.299 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:26.299 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 msgr2=0x7f669c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 0x7f669c032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f66ac0096a0 tx=0x7f66ac006b40).stop 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 msgr2=0x7f66b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f66a4014000 tx=0x7f66a400b040).stop 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 shutdown_connections 2024-01-21T07:30:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f669c02ff60 0x7f669c032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 --2- 172.21.15.115:0/759643048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66b00f6510 0x7f66b0106730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 >> 172.21.15.115:0/759643048 conn(0x7f66b00f1c70 msgr2=0x7f66b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 shutdown_connections 2024-01-21T07:30:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:26.299+0000 7f66b8580700 1 -- 172.21.15.115:0/759643048 wait complete. 2024-01-21T07:30:26.302 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:25.820921+0000 mgr.smithi107.adejht (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:27.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:26 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:26.298268+0000 mon.smithi107 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.115:0/759643048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:27.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:27.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:28.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:27.821348+0000 mgr.smithi107.adejht (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:28.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.848+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/3436381684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 msgr2=0x7fa5f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.848+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/3436381684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f40f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa5e4009fb0 tx=0x7fa5e4009b30).stop 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.848+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/3436381684 shutdown_connections 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.848+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/3436381684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f40f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.848+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/3436381684 >> 172.21.15.115:0/3436381684 conn(0x7fa5f40f0b70 msgr2=0x7fa5f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.849+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/3436381684 shutdown_connections 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.849+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/3436381684 wait complete. 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.849+0000 7fa5fb8eb700 1 Processor -- start 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.849+0000 7fa5fb8eb700 1 -- start start 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.849+0000 7fa5fb8eb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5fb8eb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4105ac0 con 0x7fa5f40f5410 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5f9687700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:28.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5f9687700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51864/0 (socket says 172.21.15.115:51864) 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5f9687700 1 -- 172.21.15.115:0/2699633682 learned_addr learned my addr 172.21.15.115:0/2699633682 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5f9687700 1 -- 172.21.15.115:0/2699633682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4005040 con 0x7fa5f40f5410 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5f9687700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa5e4013510 tx=0x7fa5e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e4009320 con 0x7fa5f40f5410 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f4105cc0 con 0x7fa5f40f5410 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5e4013f00 con 0x7fa5f40f5410 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e4014d90 con 0x7fa5f40f5410 2024-01-21T07:30:28.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.850+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4106150 con 0x7fa5f40f5410 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.851+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5e4014420 con 0x7fa5f40f5410 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.851+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f4045ad0 con 0x7fa5f40f5410 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.851+0000 7fa5e9ffb700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.851+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5e4041080 con 0x7fa5f40f5410 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.852+0000 7fa5f8e86700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:28.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.852+0000 7fa5f8e86700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 0x7fa5e0032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:28.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:28.856+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5e40095f0 con 0x7fa5f40f5410 2024-01-21T07:30:29.063 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.061+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5f4107000 con 0x7fa5f40f5410 2024-01-21T07:30:29.064 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.062+0000 7fa5e9ffb700 1 -- 172.21.15.115:0/2699633682 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa5e401db70 con 0x7fa5f40f5410 2024-01-21T07:30:29.064 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:29.064 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 msgr2=0x7fa5e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 0x7fa5e0032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa5f00096a0 tx=0x7fa5f0006b40).stop 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 msgr2=0x7fa5f41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa5e4013510 tx=0x7fa5e400b040).stop 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 shutdown_connections 2024-01-21T07:30:29.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa5e002ff60 0x7fa5e0032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:29.067 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 --2- 172.21.15.115:0/2699633682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5f40f5410 0x7fa5f41055d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:29.067 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 >> 172.21.15.115:0/2699633682 conn(0x7fa5f40f0b70 msgr2=0x7fa5f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:29.067 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 shutdown_connections 2024-01-21T07:30:29.067 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:29.064+0000 7fa5fb8eb700 1 -- 172.21.15.115:0/2699633682 wait complete. 2024-01-21T07:30:29.067 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:29.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:29.063386+0000 mon.smithi107 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.115:0/2699633682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:30.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:30.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:30.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:29.821702+0000 mgr.smithi107.adejht (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 -- 172.21.15.115:0/2713257323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 msgr2=0x7f3c300f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 --2- 172.21.15.115:0/2713257323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c300f5750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f3c20009fb0 tx=0x7f3c20009b30).stop 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 -- 172.21.15.115:0/2713257323 shutdown_connections 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 --2- 172.21.15.115:0/2713257323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c300f5750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 -- 172.21.15.115:0/2713257323 >> 172.21.15.115:0/2713257323 conn(0x7f3c300f0a90 msgr2=0x7f3c300f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 -- 172.21.15.115:0/2713257323 shutdown_connections 2024-01-21T07:30:31.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 -- 172.21.15.115:0/2713257323 wait complete. 2024-01-21T07:30:31.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.656+0000 7f3c36a05700 1 Processor -- start 2024-01-21T07:30:31.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c36a05700 1 -- start start 2024-01-21T07:30:31.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c36a05700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:31.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c36a05700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30106f10 con 0x7f3c300f5330 2024-01-21T07:30:31.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c2ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:58308/0 (socket says 172.21.15.115:58308) 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c2ffff700 1 -- 172.21.15.115:0/644360773 learned_addr learned my addr 172.21.15.115:0/644360773 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c2ffff700 1 -- 172.21.15.115:0/644360773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20005040 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.657+0000 7f3c2ffff700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3c20009f80 tx=0x7f3c2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c20009320 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c30107110 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c20013ec0 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c2001ece0 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c301075a0 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.658+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c20014420 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.659+0000 7f3c2cff9700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 0x7f3c18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.659+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c20040d10 con 0x7f3c300f5330 2024-01-21T07:30:31.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.659+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c30045ad0 con 0x7f3c300f5330 2024-01-21T07:30:31.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.659+0000 7f3c2f7fe700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 0x7f3c18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:31.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.659+0000 7f3c2f7fe700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 0x7f3c18032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3c240096a0 tx=0x7f3c24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:31.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.663+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c2001c8f0 con 0x7f3c300f5330 2024-01-21T07:30:31.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.873+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c30100e50 con 0x7f3c300f5330 2024-01-21T07:30:31.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.874+0000 7f3c2cff9700 1 -- 172.21.15.115:0/644360773 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3c2001c320 con 0x7f3c300f5330 2024-01-21T07:30:31.876 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:31.876 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 msgr2=0x7f3c18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 0x7f3c18032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3c240096a0 tx=0x7f3c24006b40).stop 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 msgr2=0x7f3c30106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3c20009f80 tx=0x7f3c2000b040).stop 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 shutdown_connections 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3c1802ff60 0x7f3c18032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 --2- 172.21.15.115:0/644360773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c300f5330 0x7f3c30106a20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.875+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 >> 172.21.15.115:0/644360773 conn(0x7f3c300f0a90 msgr2=0x7f3c300f18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.876+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 shutdown_connections 2024-01-21T07:30:31.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:31.876+0000 7f3c36a05700 1 -- 172.21.15.115:0/644360773 wait complete. 2024-01-21T07:30:31.879 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:31.821987+0000 mgr.smithi107.adejht (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:31.875107+0000 mon.smithi107 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.115:0/644360773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:33.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:33.295 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:33.822334+0000 mgr.smithi107.adejht (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- 172.21.15.115:0/2708862309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 msgr2=0x7f44d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 --2- 172.21.15.115:0/2708862309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d80f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f44c8009fb0 tx=0x7f44c8009b30).stop 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- 172.21.15.115:0/2708862309 shutdown_connections 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 --2- 172.21.15.115:0/2708862309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d80f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- 172.21.15.115:0/2708862309 >> 172.21.15.115:0/2708862309 conn(0x7f44d80f0b70 msgr2=0x7f44d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:34.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- 172.21.15.115:0/2708862309 shutdown_connections 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- 172.21.15.115:0/2708862309 wait complete. 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 Processor -- start 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.396+0000 7f44de5fb700 1 -- start start 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44de5fb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44de5fb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8105b20 con 0x7f44d80f5410 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:58332/0 (socket says 172.21.15.115:58332) 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d7fff700 1 -- 172.21.15.115:0/3236495575 learned_addr learned my addr 172.21.15.115:0/3236495575 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d7fff700 1 -- 172.21.15.115:0/3236495575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8005040 con 0x7f44d80f5410 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d7fff700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f44c8013510 tx=0x7f44c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c8009320 con 0x7f44d80f5410 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d8105d20 con 0x7f44d80f5410 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.397+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44c8013f00 con 0x7f44d80f5410 2024-01-21T07:30:34.402 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c8014d90 con 0x7f44d80f5410 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81061b0 con 0x7f44d80f5410 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44c80095f0 con 0x7f44d80f5410 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8045ad0 con 0x7f44d80f5410 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44d57fa700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 0x7f44c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.398+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44c80410c0 con 0x7f44d80f5410 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.399+0000 7f44cffff700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 0x7f44c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:34.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.399+0000 7f44cffff700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 0x7f44c4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f44c00096a0 tx=0x7f44c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:34.405 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.403+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44c8014420 con 0x7f44d80f5410 2024-01-21T07:30:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.607+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44d80fae60 con 0x7f44d80f5410 2024-01-21T07:30:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.608+0000 7f44d57fa700 1 -- 172.21.15.115:0/3236495575 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44c8010230 con 0x7f44d80f5410 2024-01-21T07:30:34.611 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:34.611 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 msgr2=0x7f44c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 0x7f44c4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f44c00096a0 tx=0x7f44c0006b40).stop 2024-01-21T07:30:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 msgr2=0x7f44d8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f44c8013510 tx=0x7f44c800b040).stop 2024-01-21T07:30:34.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 shutdown_connections 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f44c402ff60 0x7f44c4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 --2- 172.21.15.115:0/3236495575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44d80f5410 0x7f44d8105630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 >> 172.21.15.115:0/3236495575 conn(0x7f44d80f0b70 msgr2=0x7f44d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 shutdown_connections 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:34.610+0000 7f44de5fb700 1 -- 172.21.15.115:0/3236495575 wait complete. 2024-01-21T07:30:34.613 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:35.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:34.609649+0000 mon.smithi107 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.115:0/3236495575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:36.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:36.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:36.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:35.822715+0000 mgr.smithi107.adejht (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 -- 172.21.15.115:0/2469710114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 msgr2=0x7efeb00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 --2- 172.21.15.115:0/2469710114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb00f8f20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efea0009fb0 tx=0x7efea0009b30).stop 2024-01-21T07:30:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 -- 172.21.15.115:0/2469710114 shutdown_connections 2024-01-21T07:30:37.208 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 --2- 172.21.15.115:0/2469710114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb00f8f20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 -- 172.21.15.115:0/2469710114 >> 172.21.15.115:0/2469710114 conn(0x7efeb00f09a0 msgr2=0x7efeb00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 -- 172.21.15.115:0/2469710114 shutdown_connections 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.204+0000 7efeaffff700 1 -- 172.21.15.115:0/2469710114 wait complete. 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaffff700 1 Processor -- start 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaffff700 1 -- start start 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaffff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb0108c10 con 0x7efeb00601e0 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:58348/0 (socket says 172.21.15.115:58348) 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaeffd700 1 -- 172.21.15.115:0/323166821 learned_addr learned my addr 172.21.15.115:0/323166821 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:37.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaeffd700 1 -- 172.21.15.115:0/323166821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0005040 con 0x7efeb00601e0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.205+0000 7efeaeffd700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7efea0013510 tx=0x7efea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea0009320 con 0x7efeb00601e0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efea0013f00 con 0x7efeb00601e0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb0108e10 con 0x7efeb00601e0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea0014d90 con 0x7efeb00601e0 2024-01-21T07:30:37.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb01092a0 con 0x7efeb00601e0 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efea00263a0 con 0x7efeb00601e0 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.206+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0045ad0 con 0x7efeb00601e0 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.207+0000 7efe977fe700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.207+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efea00410c0 con 0x7efeb00601e0 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.207+0000 7efeae7fc700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:37.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.207+0000 7efeae7fc700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 0x7efe98032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:37.213 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.211+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efea00126f0 con 0x7efeb00601e0 2024-01-21T07:30:37.419 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.417+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efeb010a150 con 0x7efeb00601e0 2024-01-21T07:30:37.419 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.417+0000 7efe977fe700 1 -- 172.21.15.115:0/323166821 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efea0012330 con 0x7efeb00601e0 2024-01-21T07:30:37.420 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:37.420 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 msgr2=0x7efe98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 0x7efe98032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).stop 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 msgr2=0x7efeb0108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7efea0013510 tx=0x7efea000b040).stop 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 shutdown_connections 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 --2- 172.21.15.115:0/323166821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efeb00601e0 0x7efeb0108720 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 >> 172.21.15.115:0/323166821 conn(0x7efeb00f09a0 msgr2=0x7efeb00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 shutdown_connections 2024-01-21T07:30:37.422 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:37.419+0000 7efeaffff700 1 -- 172.21.15.115:0/323166821 wait complete. 2024-01-21T07:30:37.423 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:38.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:37.418824+0000 mon.smithi107 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.115:0/323166821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:38.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:38.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:39.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:37.823031+0000 mgr.smithi107.adejht (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:39.926 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 -- 172.21.15.115:0/1560681604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 msgr2=0x7fe65c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:39.926 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 --2- 172.21.15.115:0/1560681604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe64c009fb0 tx=0x7fe64c009b30).stop 2024-01-21T07:30:39.926 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 -- 172.21.15.115:0/1560681604 shutdown_connections 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 --2- 172.21.15.115:0/1560681604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 -- 172.21.15.115:0/1560681604 >> 172.21.15.115:0/1560681604 conn(0x7fe65c0f0b70 msgr2=0x7fe65c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 -- 172.21.15.115:0/1560681604 shutdown_connections 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.923+0000 7fe663705700 1 -- 172.21.15.115:0/1560681604 wait complete. 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.924+0000 7fe663705700 1 Processor -- start 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.924+0000 7fe663705700 1 -- start start 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.924+0000 7fe663705700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:39.927 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.924+0000 7fe663705700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c1049b0 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe6614a1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe6614a1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36802/0 (socket says 172.21.15.115:36802) 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe6614a1700 1 -- 172.21.15.115:0/1552782324 learned_addr learned my addr 172.21.15.115:0/1552782324 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe6614a1700 1 -- 172.21.15.115:0/1552782324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c005040 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe6614a1700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe64c013510 tx=0x7fe64c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe64c009320 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe64c013f00 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c104bb0 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe64c014d90 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.925+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c105040 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.926+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe64c014420 con 0x7fe65c0f5410 2024-01-21T07:30:39.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.926+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c045ad0 con 0x7fe65c0f5410 2024-01-21T07:30:39.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.927+0000 7fe651ffb700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 0x7fe648034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:39.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.927+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe64c040f90 con 0x7fe65c0f5410 2024-01-21T07:30:39.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.927+0000 7fe660ca0700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 0x7fe648034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:39.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.927+0000 7fe660ca0700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 0x7fe648034670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6580096a0 tx=0x7fe658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:39.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:39.931+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe64c0095f0 con 0x7fe65c0f5410 2024-01-21T07:30:40.141 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.139+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe65c105ef0 con 0x7fe65c0f5410 2024-01-21T07:30:40.141 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.139+0000 7fe651ffb700 1 -- 172.21.15.115:0/1552782324 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe64c01db70 con 0x7fe65c0f5410 2024-01-21T07:30:40.142 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:40.142 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 msgr2=0x7fe648034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 0x7fe648034670 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6580096a0 tx=0x7fe658006b40).stop 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 msgr2=0x7fe65c1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe64c013510 tx=0x7fe64c00b040).stop 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 shutdown_connections 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe6480321b0 0x7fe648034670 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 --2- 172.21.15.115:0/1552782324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe65c0f5410 0x7fe65c1044c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 >> 172.21.15.115:0/1552782324 conn(0x7fe65c0f0b70 msgr2=0x7fe65c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 shutdown_connections 2024-01-21T07:30:40.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:40.141+0000 7fe663705700 1 -- 172.21.15.115:0/1552782324 wait complete. 2024-01-21T07:30:40.145 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:40.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:39.823370+0000 mgr.smithi107.adejht (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:40.140748+0000 mon.smithi107 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.115:0/1552782324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:41.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:41.515 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:42.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 -- 172.21.15.115:0/110759943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 msgr2=0x7f46ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 --2- 172.21.15.115:0/110759943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f46dc009fb0 tx=0x7f46dc009b30).stop 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 -- 172.21.15.115:0/110759943 shutdown_connections 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 --2- 172.21.15.115:0/110759943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 -- 172.21.15.115:0/110759943 >> 172.21.15.115:0/110759943 conn(0x7f46ec0f0b70 msgr2=0x7f46ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 -- 172.21.15.115:0/110759943 shutdown_connections 2024-01-21T07:30:42.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.632+0000 7f46f276f700 1 -- 172.21.15.115:0/110759943 wait complete. 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.633+0000 7f46f276f700 1 Processor -- start 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.633+0000 7f46f276f700 1 -- start start 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.633+0000 7f46f276f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.633+0000 7f46f276f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec060240 con 0x7f46ec0f5410 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46ebfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46ebfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36818/0 (socket says 172.21.15.115:36818) 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46ebfff700 1 -- 172.21.15.115:0/1980120048 learned_addr learned my addr 172.21.15.115:0/1980120048 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:42.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46ebfff700 1 -- 172.21.15.115:0/1980120048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc005040 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46ebfff700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f46dc013510 tx=0x7f46dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46dc009320 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec060440 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46dc013f00 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46dc014d90 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.634+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec109ae0 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.635+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46dc0263a0 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.635+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec045ad0 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.635+0000 7f46e8ff9700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 0x7f46d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.635+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46dc0413d0 con 0x7f46ec0f5410 2024-01-21T07:30:42.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.635+0000 7f46eb7fe700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 0x7f46d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:42.640 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.636+0000 7f46eb7fe700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 0x7f46d4032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f46e00096a0 tx=0x7f46e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:42.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.640+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46dc014420 con 0x7f46ec0f5410 2024-01-21T07:30:42.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.851+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46ec0fae60 con 0x7f46ec0f5410 2024-01-21T07:30:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.852+0000 7f46e8ff9700 1 -- 172.21.15.115:0/1980120048 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46dc014420 con 0x7f46ec0f5410 2024-01-21T07:30:42.854 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:42.854 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 msgr2=0x7f46d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 0x7f46d4032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f46e00096a0 tx=0x7f46e0006b40).stop 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 msgr2=0x7f46ec109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f46dc013510 tx=0x7f46dc00b040).stop 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 shutdown_connections 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46d402ff60 0x7f46d4032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 --2- 172.21.15.115:0/1980120048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46ec0f5410 0x7f46ec109430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.853+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 >> 172.21.15.115:0/1980120048 conn(0x7f46ec0f0b70 msgr2=0x7f46ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.854+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 shutdown_connections 2024-01-21T07:30:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:42.854+0000 7f46f276f700 1 -- 172.21.15.115:0/1980120048 wait complete. 2024-01-21T07:30:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:43.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:41.823674+0000 mgr.smithi107.adejht (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:44.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:42.853186+0000 mon.smithi107 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.115:0/1980120048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:44.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:44.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:43.824001+0000 mgr.smithi107.adejht (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 -- 172.21.15.115:0/4111317065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 msgr2=0x7f2ab40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 --2- 172.21.15.115:0/4111317065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab40f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2aa8009fb0 tx=0x7f2aa8009b30).stop 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 -- 172.21.15.115:0/4111317065 shutdown_connections 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 --2- 172.21.15.115:0/4111317065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab40f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 -- 172.21.15.115:0/4111317065 >> 172.21.15.115:0/4111317065 conn(0x7f2ab40f1c70 msgr2=0x7f2ab40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 -- 172.21.15.115:0/4111317065 shutdown_connections 2024-01-21T07:30:45.398 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.394+0000 7f2abafdd700 1 -- 172.21.15.115:0/4111317065 wait complete. 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2abafdd700 1 Processor -- start 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2abafdd700 1 -- start start 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2abafdd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2abafdd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab4069e20 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2ab8d79700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2ab8d79700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36836/0 (socket says 172.21.15.115:36836) 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.395+0000 7f2ab8d79700 1 -- 172.21.15.115:0/73127674 learned_addr learned my addr 172.21.15.115:0/73127674 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2ab8d79700 1 -- 172.21.15.115:0/73127674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8005040 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2ab8d79700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2aa8013510 tx=0x7f2aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa8009320 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab4069fc0 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa8013f00 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa8014d90 con 0x7f2ab40f6510 2024-01-21T07:30:45.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.396+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab406a450 con 0x7f2ab40f6510 2024-01-21T07:30:45.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2aa80263a0 con 0x7f2ab40f6510 2024-01-21T07:30:45.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2ab17fa700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:45.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4059100 con 0x7f2ab40f6510 2024-01-21T07:30:45.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2aa8041370 con 0x7f2ab40f6510 2024-01-21T07:30:45.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2ab3fff700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:45.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.397+0000 7f2ab3fff700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 0x7f2aa4032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2aa00096a0 tx=0x7f2aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:45.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.401+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2aa801cad0 con 0x7f2ab40f6510 2024-01-21T07:30:45.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.608+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ab40fbf60 con 0x7f2ab40f6510 2024-01-21T07:30:45.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.608+0000 7f2ab17fa700 1 -- 172.21.15.115:0/73127674 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2aa802c090 con 0x7f2ab40f6510 2024-01-21T07:30:45.611 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:45.611 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:45.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 msgr2=0x7f2aa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 0x7f2aa4032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2aa00096a0 tx=0x7f2aa0006b40).stop 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 msgr2=0x7f2ab406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f2aa8013510 tx=0x7f2aa800b040).stop 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 shutdown_connections 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 --2- 172.21.15.115:0/73127674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab40f6510 0x7f2ab406b9a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 >> 172.21.15.115:0/73127674 conn(0x7f2ab40f1c70 msgr2=0x7f2ab40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 shutdown_connections 2024-01-21T07:30:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:45.610+0000 7f2abafdd700 1 -- 172.21.15.115:0/73127674 wait complete. 2024-01-21T07:30:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:46.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:45.610010+0000 mon.smithi107 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.115:0/73127674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:47.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:47.001 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:47.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:45.824390+0000 mgr.smithi107.adejht (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:48.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- 172.21.15.115:0/619533611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 msgr2=0x7f83140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 --2- 172.21.15.115:0/619533611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83140f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8304009fb0 tx=0x7f8304009b30).stop 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- 172.21.15.115:0/619533611 shutdown_connections 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 --2- 172.21.15.115:0/619533611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83140f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- 172.21.15.115:0/619533611 >> 172.21.15.115:0/619533611 conn(0x7f83140f09a0 msgr2=0x7f83140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- 172.21.15.115:0/619533611 shutdown_connections 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- 172.21.15.115:0/619533611 wait complete. 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 Processor -- start 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.115+0000 7f831359e700 1 -- start start 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831359e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:48.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831359e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314060220 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36848/0 (socket says 172.21.15.115:36848) 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831259c700 1 -- 172.21.15.115:0/465306983 learned_addr learned my addr 172.21.15.115:0/465306983 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831259c700 1 -- 172.21.15.115:0/465306983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304005040 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831259c700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8304013510 tx=0x7f830400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8304009320 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8314060420 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8304013f00 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8304014dc0 con 0x7f83140f51a0 2024-01-21T07:30:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.116+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8314109880 con 0x7f83140f51a0 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.117+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83040095f0 con 0x7f83140f51a0 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.117+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314059100 con 0x7f83140f51a0 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.117+0000 7f8302ffd700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 0x7f82fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.117+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8304041070 con 0x7f83140f51a0 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.118+0000 7f8311d9b700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 0x7f82fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.118+0000 7f8311d9b700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 0x7f82fc032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f83080096a0 tx=0x7f8308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:48.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.122+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f830406d2b0 con 0x7f83140f51a0 2024-01-21T07:30:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:47.824724+0000 mgr.smithi107.adejht (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:48.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.329+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8314045ad0 con 0x7f83140f51a0 2024-01-21T07:30:48.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.329+0000 7f8302ffd700 1 -- 172.21.15.115:0/465306983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8304014420 con 0x7f83140f51a0 2024-01-21T07:30:48.332 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:48.332 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 msgr2=0x7f82fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 0x7f82fc032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f83080096a0 tx=0x7f8308006b40).stop 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 msgr2=0x7f83141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8304013510 tx=0x7f830400b040).stop 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 shutdown_connections 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f82fc02ffb0 0x7f82fc032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 --2- 172.21.15.115:0/465306983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83140f51a0 0x7f83141091d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 >> 172.21.15.115:0/465306983 conn(0x7f83140f09a0 msgr2=0x7f83140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 shutdown_connections 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:48.331+0000 7f831359e700 1 -- 172.21.15.115:0/465306983 wait complete. 2024-01-21T07:30:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:48.330934+0000 mon.smithi107 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.115:0/465306983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:49.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:49.745 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:50.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:49.825032+0000 mgr.smithi107.adejht (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 -- 172.21.15.115:0/598857892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 msgr2=0x7fb8f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/598857892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f00f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb8e0009fb0 tx=0x7fb8e0009b30).stop 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 -- 172.21.15.115:0/598857892 shutdown_connections 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/598857892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f00f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 -- 172.21.15.115:0/598857892 >> 172.21.15.115:0/598857892 conn(0x7fb8f00f0b70 msgr2=0x7fb8f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:50.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 -- 172.21.15.115:0/598857892 shutdown_connections 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.895+0000 7fb8f5b41700 1 -- 172.21.15.115:0/598857892 wait complete. 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8f5b41700 1 Processor -- start 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8f5b41700 1 -- start start 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8f5b41700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8f5b41700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0108e00 con 0x7fb8f00f5410 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8ef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8ef7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35296/0 (socket says 172.21.15.115:35296) 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.896+0000 7fb8ef7fe700 1 -- 172.21.15.115:0/4281963285 learned_addr learned my addr 172.21.15.115:0/4281963285 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8ef7fe700 1 -- 172.21.15.115:0/4281963285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0005040 con 0x7fb8f00f5410 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8ef7fe700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fb8e0013510 tx=0x7fb8e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e0009320 con 0x7fb8f00f5410 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f0109000 con 0x7fb8f00f5410 2024-01-21T07:30:50.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8e0013f00 con 0x7fb8f00f5410 2024-01-21T07:30:50.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e0014d90 con 0x7fb8f00f5410 2024-01-21T07:30:50.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.897+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f0109490 con 0x7fb8f00f5410 2024-01-21T07:30:50.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8e0014420 con 0x7fb8f00f5410 2024-01-21T07:30:50.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0045ad0 con 0x7fb8f00f5410 2024-01-21T07:30:50.902 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8d7fff700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:50.902 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8e0041000 con 0x7fb8f00f5410 2024-01-21T07:30:50.902 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8eeffd700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:50.902 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.898+0000 7fb8eeffd700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb8e40096a0 tx=0x7fb8e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:50.904 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:50.903+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e006d2b0 con 0x7fb8f00f5410 2024-01-21T07:30:51.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.107+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8f00faf50 con 0x7fb8f00f5410 2024-01-21T07:30:51.110 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.108+0000 7fb8d7fff700 1 -- 172.21.15.115:0/4281963285 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8e001dd50 con 0x7fb8f00f5410 2024-01-21T07:30:51.110 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:51.110 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:51.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 msgr2=0x7fb8d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb8e40096a0 tx=0x7fb8e4006b40).stop 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 msgr2=0x7fb8f0108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fb8e0013510 tx=0x7fb8e000b040).stop 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 shutdown_connections 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 --2- 172.21.15.115:0/4281963285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00f5410 0x7fb8f0108910 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 >> 172.21.15.115:0/4281963285 conn(0x7fb8f00f0b70 msgr2=0x7fb8f005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.109+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 shutdown_connections 2024-01-21T07:30:51.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:51.110+0000 7fb8f5b41700 1 -- 172.21.15.115:0/4281963285 wait complete. 2024-01-21T07:30:51.113 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:51.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:51.109332+0000 mon.smithi107 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.115:0/4281963285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:52.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:52.512 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:53.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:51.825382+0000 mgr.smithi107.adejht (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:53.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- 172.21.15.115:0/2190082919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 msgr2=0x7efea40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:53.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 --2- 172.21.15.115:0/2190082919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea40f66c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7efe98009fb0 tx=0x7efe98009b30).stop 2024-01-21T07:30:53.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- 172.21.15.115:0/2190082919 shutdown_connections 2024-01-21T07:30:53.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 --2- 172.21.15.115:0/2190082919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea40f66c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:53.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- 172.21.15.115:0/2190082919 >> 172.21.15.115:0/2190082919 conn(0x7efea40f1ac0 msgr2=0x7efea40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:53.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- 172.21.15.115:0/2190082919 shutdown_connections 2024-01-21T07:30:53.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- 172.21.15.115:0/2190082919 wait complete. 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 Processor -- start 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.609+0000 7efea9cbb700 1 -- start start 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea9cbb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea9cbb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea4106990 con 0x7efea40f62a0 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea8cb9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea8cb9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35308/0 (socket says 172.21.15.115:35308) 2024-01-21T07:30:53.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea8cb9700 1 -- 172.21.15.115:0/1192172770 learned_addr learned my addr 172.21.15.115:0/1192172770 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea8cb9700 1 -- 172.21.15.115:0/1192172770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98005040 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea8cb9700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7efe98013510 tx=0x7efe9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.610+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe98009320 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea4106b90 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4107020 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe98013f00 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe98014dc0 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe980095f0 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.611+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea4045ad0 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.612+0000 7efea17fa700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 0x7efe94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.612+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe98041140 con 0x7efea40f62a0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.612+0000 7efea3fff700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 0x7efe94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:53.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.612+0000 7efea3fff700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 0x7efe94032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efe900096a0 tx=0x7efe90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:53.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.617+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe98014420 con 0x7efea40f62a0 2024-01-21T07:30:53.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.827+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efea40fbce0 con 0x7efea40f62a0 2024-01-21T07:30:53.830 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.829+0000 7efea17fa700 1 -- 172.21.15.115:0/1192172770 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe98010230 con 0x7efea40f62a0 2024-01-21T07:30:53.830 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:53.830 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.829+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 msgr2=0x7efe94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.829+0000 7efea9cbb700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 0x7efe94032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efe900096a0 tx=0x7efe90006b40).stop 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.829+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 msgr2=0x7efea41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.829+0000 7efea9cbb700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7efe98013510 tx=0x7efe9800b040).stop 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 shutdown_connections 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe9402ff60 0x7efe94032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 --2- 172.21.15.115:0/1192172770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efea40f62a0 0x7efea41064a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 >> 172.21.15.115:0/1192172770 conn(0x7efea40f1ac0 msgr2=0x7efea40f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 shutdown_connections 2024-01-21T07:30:53.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:53.830+0000 7efea9cbb700 1 -- 172.21.15.115:0/1192172770 wait complete. 2024-01-21T07:30:53.833 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:53.825716+0000 mgr.smithi107.adejht (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:53.829267+0000 mon.smithi107 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.115:0/1192172770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:55.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:55.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:55.825985+0000 mgr.smithi107.adejht (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:56.407 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- 172.21.15.115:0/812308593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 msgr2=0x7fe4c80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 --2- 172.21.15.115:0/812308593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f7d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe4bc009fb0 tx=0x7fe4bc009b30).stop 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- 172.21.15.115:0/812308593 shutdown_connections 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 --2- 172.21.15.115:0/812308593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f7d00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- 172.21.15.115:0/812308593 >> 172.21.15.115:0/812308593 conn(0x7fe4c80f0120 msgr2=0x7fe4c80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- 172.21.15.115:0/812308593 shutdown_connections 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- 172.21.15.115:0/812308593 wait complete. 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 Processor -- start 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.404+0000 7fe4ce397700 1 -- start start 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4ce397700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4ce397700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80f5630 con 0x7fe4c80f78e0 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4cd395700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4cd395700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35320/0 (socket says 172.21.15.115:35320) 2024-01-21T07:30:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4cd395700 1 -- 172.21.15.115:0/1196330073 learned_addr learned my addr 172.21.15.115:0/1196330073 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4cd395700 1 -- 172.21.15.115:0/1196330073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc005040 con 0x7fe4c80f78e0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.405+0000 7fe4cd395700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe4bc009830 tx=0x7fe4bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.406+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4bc009320 con 0x7fe4c80f78e0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.406+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c80f57d0 con 0x7fe4c80f78e0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.406+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4bc0136c0 con 0x7fe4c80f78e0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.406+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4bc01ee90 con 0x7fe4c80f78e0 2024-01-21T07:30:56.409 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.406+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c80f5c60 con 0x7fe4c80f78e0 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4bc014420 con 0x7fe4c80f78e0 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4b9ffb700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 0x7fe4b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4bc040f50 con 0x7fe4c80f78e0 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c8059100 con 0x7fe4c80f78e0 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4ccb94700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 0x7fe4b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:56.410 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.407+0000 7fe4ccb94700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 0x7fe4b40323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe4c40096a0 tx=0x7fe4c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:56.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.412+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4bc01c910 con 0x7fe4c80f78e0 2024-01-21T07:30:56.620 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.619+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4c80fbd80 con 0x7fe4c80f78e0 2024-01-21T07:30:56.620 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.619+0000 7fe4b9ffb700 1 -- 172.21.15.115:0/1196330073 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4bc01c320 con 0x7fe4c80f78e0 2024-01-21T07:30:56.620 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:56.621 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:56.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 msgr2=0x7fe4b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:56.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 0x7fe4b40323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe4c40096a0 tx=0x7fe4c4006b40).stop 2024-01-21T07:30:56.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 msgr2=0x7fe4c80f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:56.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe4bc009830 tx=0x7fe4bc00b040).stop 2024-01-21T07:30:56.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 shutdown_connections 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe4b402ff10 0x7fe4b40323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 --2- 172.21.15.115:0/1196330073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c80f78e0 0x7fe4c80f71b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 >> 172.21.15.115:0/1196330073 conn(0x7fe4c80f0120 msgr2=0x7fe4c80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 shutdown_connections 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:56.621+0000 7fe4ce397700 1 -- 172.21.15.115:0/1196330073 wait complete. 2024-01-21T07:30:56.623 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:57.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:56 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:56.619693+0000 mon.smithi107 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.115:0/1196330073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:30:58.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:30:58.029 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:30:58.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:57.826289+0000 mgr.smithi107.adejht (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 -- 172.21.15.115:0/266977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 msgr2=0x7fe3f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 --2- 172.21.15.115:0/266977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f40f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe3e4009fb0 tx=0x7fe3e4009b30).stop 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 -- 172.21.15.115:0/266977330 shutdown_connections 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 --2- 172.21.15.115:0/266977330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f40f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 -- 172.21.15.115:0/266977330 >> 172.21.15.115:0/266977330 conn(0x7fe3f40f0b70 msgr2=0x7fe3f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 -- 172.21.15.115:0/266977330 shutdown_connections 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.120+0000 7fe3faa21700 1 -- 172.21.15.115:0/266977330 wait complete. 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3faa21700 1 Processor -- start 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3faa21700 1 -- start start 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3faa21700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3faa21700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f4060280 con 0x7fe3f40f5410 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3f3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3f3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35340/0 (socket says 172.21.15.115:35340) 2024-01-21T07:30:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.121+0000 7fe3f3fff700 1 -- 172.21.15.115:0/3501580585 learned_addr learned my addr 172.21.15.115:0/3501580585 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3f3fff700 1 -- 172.21.15.115:0/3501580585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e4005040 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3f3fff700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe3e4013510 tx=0x7fe3e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e4009320 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f4060480 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3e4013f00 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e4014d90 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.122+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f4109b20 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.123+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3e4014420 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.123+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f4045ad0 con 0x7fe3f40f5410 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.123+0000 7fe3f0ff9700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:30:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.123+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3e4041000 con 0x7fe3f40f5410 2024-01-21T07:30:59.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.123+0000 7fe3f37fe700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:30:59.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.124+0000 7fe3f37fe700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe3e80096a0 tx=0x7fe3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:30:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.128+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3e40095f0 con 0x7fe3f40f5410 2024-01-21T07:30:59.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.336+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3f410a340 con 0x7fe3f40f5410 2024-01-21T07:30:59.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.336+0000 7fe3f0ff9700 1 -- 172.21.15.115:0/3501580585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe3e401db70 con 0x7fe3f40f5410 2024-01-21T07:30:59.338 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:30:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 msgr2=0x7fe3dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe3e80096a0 tx=0x7fe3e8006b40).stop 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 msgr2=0x7fe3f41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe3e4013510 tx=0x7fe3e400b040).stop 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 shutdown_connections 2024-01-21T07:30:59.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:59.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 --2- 172.21.15.115:0/3501580585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3f40f5410 0x7fe3f41093f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:30:59.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 >> 172.21.15.115:0/3501580585 conn(0x7fe3f40f0b70 msgr2=0x7fe3f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:30:59.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 shutdown_connections 2024-01-21T07:30:59.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:30:59.338+0000 7fe3faa21700 1 -- 172.21.15.115:0/3501580585 wait complete. 2024-01-21T07:30:59.340 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:30:59.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:30:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:30:59.336838+0000 mon.smithi107 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.115:0/3501580585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:00.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:00.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:00.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:30:59.826581+0000 mgr.smithi107.adejht (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 -- 172.21.15.115:0/4198602247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 msgr2=0x7f63d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 --2- 172.21.15.115:0/4198602247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d00f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f63c0009fb0 tx=0x7f63c0009b30).stop 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 -- 172.21.15.115:0/4198602247 shutdown_connections 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 --2- 172.21.15.115:0/4198602247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d00f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 -- 172.21.15.115:0/4198602247 >> 172.21.15.115:0/4198602247 conn(0x7f63d00f1c70 msgr2=0x7f63d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 -- 172.21.15.115:0/4198602247 shutdown_connections 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.867+0000 7f63d764b700 1 -- 172.21.15.115:0/4198602247 wait complete. 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d764b700 1 Processor -- start 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d764b700 1 -- start start 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d764b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d764b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0108dd0 con 0x7f63d00f6510 2024-01-21T07:31:01.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d53e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:01.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d53e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45762/0 (socket says 172.21.15.115:45762) 2024-01-21T07:31:01.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.868+0000 7f63d53e7700 1 -- 172.21.15.115:0/2746944373 learned_addr learned my addr 172.21.15.115:0/2746944373 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:01.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63d53e7700 1 -- 172.21.15.115:0/2746944373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0005040 con 0x7f63d00f6510 2024-01-21T07:31:01.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63d53e7700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f63c0014000 tx=0x7f63c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:01.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c00095f0 con 0x7f63d00f6510 2024-01-21T07:31:01.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63c0007e70 con 0x7f63d00f6510 2024-01-21T07:31:01.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0108fd0 con 0x7f63d00f6510 2024-01-21T07:31:01.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c0014ea0 con 0x7f63d00f6510 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.869+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0109460 con 0x7f63d00f6510 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.870+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63c0025070 con 0x7f63d00f6510 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.870+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0045ad0 con 0x7f63d00f6510 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.870+0000 7f63c5ffb700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 0x7f63bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.870+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63c0040930 con 0x7f63d00f6510 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.870+0000 7f63d4be6700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 0x7f63bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:01.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.871+0000 7f63d4be6700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 0x7f63bc032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f63cc0096a0 tx=0x7f63cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:01.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:01.875+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63c00079c0 con 0x7f63d00f6510 2024-01-21T07:31:02.084 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.083+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63d010aa80 con 0x7f63d00f6510 2024-01-21T07:31:02.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.084+0000 7f63c5ffb700 1 -- 172.21.15.115:0/2746944373 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63c000f300 con 0x7f63d00f6510 2024-01-21T07:31:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:02.085 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:02.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 msgr2=0x7f63bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:02.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 0x7f63bc032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f63cc0096a0 tx=0x7f63cc006b40).stop 2024-01-21T07:31:02.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 msgr2=0x7f63d01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f63c0014000 tx=0x7f63c000b040).stop 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 shutdown_connections 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f63bc02ff60 0x7f63bc032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 --2- 172.21.15.115:0/2746944373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d00f6510 0x7f63d01088e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 >> 172.21.15.115:0/2746944373 conn(0x7f63d00f1c70 msgr2=0x7f63d00f4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 shutdown_connections 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:02.085+0000 7f63d764b700 1 -- 172.21.15.115:0/2746944373 wait complete. 2024-01-21T07:31:02.087 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:01.826887+0000 mgr.smithi107.adejht (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:02.084304+0000 mon.smithi107 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.115:0/2746944373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:03.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:03.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:04.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:03.827267+0000 mgr.smithi107.adejht (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:04.596 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- 172.21.15.115:0/1803004425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 msgr2=0x7f20080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:04.596 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 --2- 172.21.15.115:0/1803004425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f20080f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1ffc009fb0 tx=0x7f1ffc009b30).stop 2024-01-21T07:31:04.596 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- 172.21.15.115:0/1803004425 shutdown_connections 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 --2- 172.21.15.115:0/1803004425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f20080f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- 172.21.15.115:0/1803004425 >> 172.21.15.115:0/1803004425 conn(0x7f20080f1c70 msgr2=0x7f20080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- 172.21.15.115:0/1803004425 shutdown_connections 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- 172.21.15.115:0/1803004425 wait complete. 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 Processor -- start 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.592+0000 7f200f0a5700 1 -- start start 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200f0a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200f0a5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2008106c20 con 0x7f20080f6510 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200ce41700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:04.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200ce41700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45776/0 (socket says 172.21.15.115:45776) 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200ce41700 1 -- 172.21.15.115:0/294453923 learned_addr learned my addr 172.21.15.115:0/294453923 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200ce41700 1 -- 172.21.15.115:0/294453923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc005040 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.593+0000 7f200ce41700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1ffc014000 tx=0x7f1ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ffc0095f0 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2008106e20 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ffc007e70 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ffc014ea0 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20081072b0 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ffc025070 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.594+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008045ad0 con 0x7f20080f6510 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.595+0000 7f20057fa700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 0x7f1ff8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:04.598 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.595+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ffc0409a0 con 0x7f20080f6510 2024-01-21T07:31:04.599 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.595+0000 7f2007fff700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 0x7f1ff8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:04.599 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.595+0000 7f2007fff700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 0x7f1ff8032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1ff40096a0 tx=0x7f1ff4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:04.600 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.599+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ffc0079c0 con 0x7f20080f6510 2024-01-21T07:31:04.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.807+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20080fbf60 con 0x7f20080f6510 2024-01-21T07:31:04.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.808+0000 7f20057fa700 1 -- 172.21.15.115:0/294453923 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ffc00f300 con 0x7f20080f6510 2024-01-21T07:31:04.809 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:04.810 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 msgr2=0x7f1ff8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 0x7f1ff8032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1ff40096a0 tx=0x7f1ff4006b40).stop 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 msgr2=0x7f2008106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1ffc014000 tx=0x7f1ffc00b040).stop 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 shutdown_connections 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1ff802ffb0 0x7f1ff8032470 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 --2- 172.21.15.115:0/294453923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20080f6510 0x7f2008106730 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:04.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 >> 172.21.15.115:0/294453923 conn(0x7f20080f1c70 msgr2=0x7f20080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:04.812 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 shutdown_connections 2024-01-21T07:31:04.812 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:04.810+0000 7f200f0a5700 1 -- 172.21.15.115:0/294453923 wait complete. 2024-01-21T07:31:04.812 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:04.808456+0000 mon.smithi107 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.115:0/294453923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:06.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:06.188 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:05.827485+0000 mgr.smithi107.adejht (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:07.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.290+0000 7f5743065700 1 -- 172.21.15.115:0/3951367774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 msgr2=0x7f573c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.290+0000 7f5743065700 1 --2- 172.21.15.115:0/3951367774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f5730009fb0 tx=0x7f5730009b30).stop 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.290+0000 7f5743065700 1 -- 172.21.15.115:0/3951367774 shutdown_connections 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.290+0000 7f5743065700 1 --2- 172.21.15.115:0/3951367774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.290+0000 7f5743065700 1 -- 172.21.15.115:0/3951367774 >> 172.21.15.115:0/3951367774 conn(0x7f573c0f0b70 msgr2=0x7f573c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 -- 172.21.15.115:0/3951367774 shutdown_connections 2024-01-21T07:31:07.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 -- 172.21.15.115:0/3951367774 wait complete. 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 Processor -- start 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 -- start start 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.291+0000 7f5743065700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c107240 con 0x7f573c0f5410 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5740e01700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5740e01700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45802/0 (socket says 172.21.15.115:45802) 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5740e01700 1 -- 172.21.15.115:0/4165041669 learned_addr learned my addr 172.21.15.115:0/4165041669 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5740e01700 1 -- 172.21.15.115:0/4165041669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730005040 con 0x7f573c0f5410 2024-01-21T07:31:07.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5740e01700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5730013510 tx=0x7f573000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730009320 con 0x7f573c0f5410 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c060220 con 0x7f573c0f5410 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5730013f00 con 0x7f573c0f5410 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730014d90 con 0x7f573c0f5410 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.292+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c10bd40 con 0x7f573c0f5410 2024-01-21T07:31:07.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.293+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57300263a0 con 0x7f573c0f5410 2024-01-21T07:31:07.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.293+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c045ad0 con 0x7f573c0f5410 2024-01-21T07:31:07.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.293+0000 7f57397fa700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 0x7f572c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:07.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.293+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5730041380 con 0x7f573c0f5410 2024-01-21T07:31:07.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.294+0000 7f573bfff700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 0x7f572c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:07.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.294+0000 7f573bfff700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 0x7f572c032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:07.301 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.298+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f573001cb50 con 0x7f573c0f5410 2024-01-21T07:31:07.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.505+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f573c0fae60 con 0x7f573c0f5410 2024-01-21T07:31:07.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.506+0000 7f57397fa700 1 -- 172.21.15.115:0/4165041669 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f573001c970 con 0x7f573c0f5410 2024-01-21T07:31:07.508 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:07.508 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:07.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 msgr2=0x7f572c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:07.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 0x7f572c032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).stop 2024-01-21T07:31:07.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 msgr2=0x7f573c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:07.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5730013510 tx=0x7f573000b040).stop 2024-01-21T07:31:07.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 shutdown_connections 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f572c02ffb0 0x7f572c032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 --2- 172.21.15.115:0/4165041669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f573c0f5410 0x7f573c10b850 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 >> 172.21.15.115:0/4165041669 conn(0x7f573c0f0b70 msgr2=0x7f573c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 shutdown_connections 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:07.508+0000 7f5743065700 1 -- 172.21.15.115:0/4165041669 wait complete. 2024-01-21T07:31:07.511 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:08.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:07.507383+0000 mon.smithi107 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.115:0/4165041669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:08.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:08.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:09.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:08 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:07.827819+0000 mgr.smithi107.adejht (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:10.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- 172.21.15.115:0/4008054275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 msgr2=0x7efe900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:10.046 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 --2- 172.21.15.115:0/4008054275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe900f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7efe8c009fb0 tx=0x7efe8c009b30).stop 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- 172.21.15.115:0/4008054275 shutdown_connections 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 --2- 172.21.15.115:0/4008054275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe900f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- 172.21.15.115:0/4008054275 >> 172.21.15.115:0/4008054275 conn(0x7efe900f0b70 msgr2=0x7efe900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- 172.21.15.115:0/4008054275 shutdown_connections 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- 172.21.15.115:0/4008054275 wait complete. 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 Processor -- start 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- start start 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.043+0000 7efe975e3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90106c20 con 0x7efe900f5410 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe9537f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe9537f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53650/0 (socket says 172.21.15.115:53650) 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe9537f700 1 -- 172.21.15.115:0/3751375101 learned_addr learned my addr 172.21.15.115:0/3751375101 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe9537f700 1 -- 172.21.15.115:0/3751375101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c005040 con 0x7efe900f5410 2024-01-21T07:31:10.047 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe9537f700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7efe8c013510 tx=0x7efe8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:10.048 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe8c009320 con 0x7efe900f5410 2024-01-21T07:31:10.048 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe90106e20 con 0x7efe900f5410 2024-01-21T07:31:10.048 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe8c013f00 con 0x7efe900f5410 2024-01-21T07:31:10.048 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe8c014d90 con 0x7efe900f5410 2024-01-21T07:31:10.048 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.044+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe901072b0 con 0x7efe900f5410 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.045+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe8c0263a0 con 0x7efe900f5410 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.045+0000 7efe85ffb700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 0x7efe7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.045+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe8c0410c0 con 0x7efe900f5410 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.045+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe90045ad0 con 0x7efe900f5410 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.046+0000 7efe94b7e700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 0x7efe7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.046+0000 7efe94b7e700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 0x7efe7c032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efe800096a0 tx=0x7efe80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:10.052 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.050+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe8c012920 con 0x7efe900f5410 2024-01-21T07:31:10.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.256+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe900fbf90 con 0x7efe900f5410 2024-01-21T07:31:10.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.257+0000 7efe85ffb700 1 -- 172.21.15.115:0/3751375101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe8c02b020 con 0x7efe900f5410 2024-01-21T07:31:10.259 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:10.259 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:10.260 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 msgr2=0x7efe7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 0x7efe7c032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efe800096a0 tx=0x7efe80006b40).stop 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 msgr2=0x7efe90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7efe8c013510 tx=0x7efe8c00b040).stop 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 shutdown_connections 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efe7c02ff60 0x7efe7c032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 --2- 172.21.15.115:0/3751375101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe900f5410 0x7efe90106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 >> 172.21.15.115:0/3751375101 conn(0x7efe900f0b70 msgr2=0x7efe900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 shutdown_connections 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:10.258+0000 7efe975e3700 1 -- 172.21.15.115:0/3751375101 wait complete. 2024-01-21T07:31:10.261 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:09.828078+0000 mgr.smithi107.adejht (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:09.946152+0000 mon.smithi107 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:31:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:09.947802+0000 mon.smithi107 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:31:11.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:10.258128+0000 mon.smithi107 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.115:0/3751375101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:11.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:11.682 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:12.760 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 -- 172.21.15.115:0/3617148163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 msgr2=0x7fc8800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 --2- 172.21.15.115:0/3617148163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc8800f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc874009fb0 tx=0x7fc874009b30).stop 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 -- 172.21.15.115:0/3617148163 shutdown_connections 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 --2- 172.21.15.115:0/3617148163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc8800f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 -- 172.21.15.115:0/3617148163 >> 172.21.15.115:0/3617148163 conn(0x7fc8800f1c70 msgr2=0x7fc8800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 -- 172.21.15.115:0/3617148163 shutdown_connections 2024-01-21T07:31:12.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.758+0000 7fc884a24700 1 -- 172.21.15.115:0/3617148163 wait complete. 2024-01-21T07:31:12.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc884a24700 1 Processor -- start 2024-01-21T07:31:12.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc884a24700 1 -- start start 2024-01-21T07:31:12.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc884a24700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:12.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc884a24700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc880106c20 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc87e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc87e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53666/0 (socket says 172.21.15.115:53666) 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.759+0000 7fc87e59c700 1 -- 172.21.15.115:0/3409639196 learned_addr learned my addr 172.21.15.115:0/3409639196 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc87e59c700 1 -- 172.21.15.115:0/3409639196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874005040 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc87e59c700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc874013510 tx=0x7fc87400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc874009320 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc880106e20 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc874013f00 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc874014d90 con 0x7fc8800f6510 2024-01-21T07:31:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.760+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8801072b0 con 0x7fc8800f6510 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8740095f0 con 0x7fc8800f6510 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc880045ad0 con 0x7fc8800f6510 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc86effd700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc874041000 con 0x7fc8800f6510 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc87dd9b700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.761+0000 7fc87dd9b700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 0x7fc868032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc8700096a0 tx=0x7fc870006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:12.768 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.766+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc874014420 con 0x7fc8800f6510 2024-01-21T07:31:12.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.972+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8800fbf60 con 0x7fc8800f6510 2024-01-21T07:31:12.974 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.972+0000 7fc86effd700 1 -- 172.21.15.115:0/3409639196 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc874010230 con 0x7fc8800f6510 2024-01-21T07:31:12.975 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:12.975 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:12.976 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 msgr2=0x7fc868032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 0x7fc868032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc8700096a0 tx=0x7fc870006b40).stop 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 msgr2=0x7fc880106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc874013510 tx=0x7fc87400b040).stop 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 shutdown_connections 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 --2- 172.21.15.115:0/3409639196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8800f6510 0x7fc880106730 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 >> 172.21.15.115:0/3409639196 conn(0x7fc8800f1c70 msgr2=0x7fc8800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 shutdown_connections 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:12.974+0000 7fc884a24700 1 -- 172.21.15.115:0/3409639196 wait complete. 2024-01-21T07:31:12.977 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:13.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:11.828354+0000 mgr.smithi107.adejht (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:14.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:12.973651+0000 mon.smithi107 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.115:0/3409639196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:14.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:14.361 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:15.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:13.828611+0000 mgr.smithi107.adejht (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:15.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.507+0000 7fa4c215f700 1 -- 172.21.15.115:0/485499716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 msgr2=0x7fa4bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:15.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.507+0000 7fa4c215f700 1 --2- 172.21.15.115:0/485499716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc0f66c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa4ac009fb0 tx=0x7fa4ac009b30).stop 2024-01-21T07:31:15.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.507+0000 7fa4c215f700 1 -- 172.21.15.115:0/485499716 shutdown_connections 2024-01-21T07:31:15.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.507+0000 7fa4c215f700 1 --2- 172.21.15.115:0/485499716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc0f66c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:15.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.507+0000 7fa4c215f700 1 -- 172.21.15.115:0/485499716 >> 172.21.15.115:0/485499716 conn(0x7fa4bc0f1aa0 msgr2=0x7fa4bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 -- 172.21.15.115:0/485499716 shutdown_connections 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 -- 172.21.15.115:0/485499716 wait complete. 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 Processor -- start 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 -- start start 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:15.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c215f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc108ba0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c115d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:15.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c115d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53682/0 (socket says 172.21.15.115:53682) 2024-01-21T07:31:15.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.508+0000 7fa4c115d700 1 -- 172.21.15.115:0/1282634264 learned_addr learned my addr 172.21.15.115:0/1282634264 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:15.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4c115d700 1 -- 172.21.15.115:0/1282634264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac005040 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4c115d700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa4ac013510 tx=0x7fa4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ac009320 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc108da0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4ac013f00 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ac014dc0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.509+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc109230 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.510+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4ac01d3d0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.510+0000 7fa4b1ffb700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 0x7fa4a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:15.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.510+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc045ad0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.510+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4ac041300 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.510+0000 7fa4c095c700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 0x7fa4a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:15.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.511+0000 7fa4c095c700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 0x7fa4a8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa4b80096a0 tx=0x7fa4b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:15.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.515+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4ac012920 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.726 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.724+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4bc10a0e0 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.727 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.725+0000 7fa4b1ffb700 1 -- 172.21.15.115:0/1282634264 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa4ac02b030 con 0x7fa4bc0f62a0 2024-01-21T07:31:15.727 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:15.727 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 msgr2=0x7fa4a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 0x7fa4a8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa4b80096a0 tx=0x7fa4b8006b40).stop 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 msgr2=0x7fa4bc1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa4ac013510 tx=0x7fa4ac00b040).stop 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 shutdown_connections 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa4a802ff60 0x7fa4a8032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 --2- 172.21.15.115:0/1282634264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4bc0f62a0 0x7fa4bc1086b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.726+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 >> 172.21.15.115:0/1282634264 conn(0x7fa4bc0f1aa0 msgr2=0x7fa4bc0f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:15.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.727+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 shutdown_connections 2024-01-21T07:31:15.730 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:15.727+0000 7fa4c215f700 1 -- 172.21.15.115:0/1282634264 wait complete. 2024-01-21T07:31:15.730 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:16.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:15.726345+0000 mon.smithi107 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.115:0/1282634264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:17.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:15.828877+0000 mgr.smithi107.adejht (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:17.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:17.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- 172.21.15.115:0/4241162763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 msgr2=0x7f36340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 --2- 172.21.15.115:0/4241162763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f36340f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3624009fb0 tx=0x7f3624009b30).stop 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- 172.21.15.115:0/4241162763 shutdown_connections 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 --2- 172.21.15.115:0/4241162763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f36340f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- 172.21.15.115:0/4241162763 >> 172.21.15.115:0/4241162763 conn(0x7f36340f0b70 msgr2=0x7f36340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- 172.21.15.115:0/4241162763 shutdown_connections 2024-01-21T07:31:18.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- 172.21.15.115:0/4241162763 wait complete. 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 Processor -- start 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.270+0000 7f3639068700 1 -- start start 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3639068700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3639068700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3634060240 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3632d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3632d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53700/0 (socket says 172.21.15.115:53700) 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3632d9d700 1 -- 172.21.15.115:0/1613568504 learned_addr learned my addr 172.21.15.115:0/1613568504 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3632d9d700 1 -- 172.21.15.115:0/1613568504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624005040 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3632d9d700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3624013510 tx=0x7f362400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3624009320 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3634060440 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3624013f00 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.271+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3624014d90 con 0x7f36340f5410 2024-01-21T07:31:18.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.272+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3634109ae0 con 0x7f36340f5410 2024-01-21T07:31:18.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.272+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36240095f0 con 0x7f36340f5410 2024-01-21T07:31:18.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.272+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634045ad0 con 0x7f36340f5410 2024-01-21T07:31:18.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.272+0000 7f361b7fe700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 0x7f361c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:18.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.272+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3624040fe0 con 0x7f36340f5410 2024-01-21T07:31:18.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.273+0000 7f363259c700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 0x7f361c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:18.277 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.273+0000 7f363259c700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 0x7f361c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f36280096a0 tx=0x7f3628006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:18.279 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.277+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f362401d7f0 con 0x7f36340f5410 2024-01-21T07:31:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:17.829112+0000 mgr.smithi107.adejht (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:18.487 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.485+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36340fae60 con 0x7f36340f5410 2024-01-21T07:31:18.488 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.486+0000 7f361b7fe700 1 -- 172.21.15.115:0/1613568504 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3624010230 con 0x7f36340f5410 2024-01-21T07:31:18.488 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:18.491 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 msgr2=0x7f361c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 0x7f361c032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f36280096a0 tx=0x7f3628006b40).stop 2024-01-21T07:31:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 msgr2=0x7f3634109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:18.492 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3624013510 tx=0x7f362400b040).stop 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 shutdown_connections 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f361c02ff60 0x7f361c032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 --2- 172.21.15.115:0/1613568504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36340f5410 0x7f3634109430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 >> 172.21.15.115:0/1613568504 conn(0x7f36340f0b70 msgr2=0x7f36340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 shutdown_connections 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:18.487+0000 7f3639068700 1 -- 172.21.15.115:0/1613568504 wait complete. 2024-01-21T07:31:18.493 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:19.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:18.487216+0000 mon.smithi107 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.115:0/1613568504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:19.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:19.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:20.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:19.829464+0000 mgr.smithi107.adejht (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:21.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.032+0000 7f32c10fd700 1 -- 172.21.15.115:0/3281399878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 msgr2=0x7f32bc0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:21.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.032+0000 7f32c10fd700 1 --2- 172.21.15.115:0/3281399878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc0fb570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f32ac009fb0 tx=0x7f32ac009b30).stop 2024-01-21T07:31:21.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 -- 172.21.15.115:0/3281399878 shutdown_connections 2024-01-21T07:31:21.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 --2- 172.21.15.115:0/3281399878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc0fb570 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:21.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 -- 172.21.15.115:0/3281399878 >> 172.21.15.115:0/3281399878 conn(0x7f32bc0f0b70 msgr2=0x7f32bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:21.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 -- 172.21.15.115:0/3281399878 shutdown_connections 2024-01-21T07:31:21.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 -- 172.21.15.115:0/3281399878 wait complete. 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 Processor -- start 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.033+0000 7f32c10fd700 1 -- start start 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32c10fd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32c10fd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc108dc0 con 0x7f32bc060200 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32bad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32bad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44412/0 (socket says 172.21.15.115:44412) 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32bad9d700 1 -- 172.21.15.115:0/2680891538 learned_addr learned my addr 172.21.15.115:0/2680891538 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32bad9d700 1 -- 172.21.15.115:0/2680891538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac005040 con 0x7f32bc060200 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32bad9d700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f32ac013510 tx=0x7f32ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:21.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ac009320 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32bc108fc0 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32ac013f00 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ac014dc0 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.034+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32bc109450 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.035+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32ac01d3d0 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.035+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32bc045ad0 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.035+0000 7f32a37fe700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 0x7f32a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.035+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32ac0413f0 con 0x7f32bc060200 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.036+0000 7f32ba59c700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 0x7f32a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:21.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.036+0000 7f32ba59c700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 0x7f32a4032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f32b00096a0 tx=0x7f32b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:21.042 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.040+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32ac012720 con 0x7f32bc060200 2024-01-21T07:31:21.249 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.247+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32bc10a300 con 0x7f32bc060200 2024-01-21T07:31:21.249 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.247+0000 7f32a37fe700 1 -- 172.21.15.115:0/2680891538 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f32ac012330 con 0x7f32bc060200 2024-01-21T07:31:21.250 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:21.250 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 msgr2=0x7f32a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 0x7f32a4032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f32b00096a0 tx=0x7f32b0006b40).stop 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 msgr2=0x7f32bc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f32ac013510 tx=0x7f32ac00b040).stop 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 shutdown_connections 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f32a402ff60 0x7f32a4032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 --2- 172.21.15.115:0/2680891538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32bc060200 0x7f32bc1088d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 >> 172.21.15.115:0/2680891538 conn(0x7f32bc0f0b70 msgr2=0x7f32bc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:21.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 shutdown_connections 2024-01-21T07:31:21.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:21.249+0000 7f32c10fd700 1 -- 172.21.15.115:0/2680891538 wait complete. 2024-01-21T07:31:21.252 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:21.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:21.248744+0000 mon.smithi107 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.115:0/2680891538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:22.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:22.635 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:23.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:21.829802+0000 mgr.smithi107.adejht (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.744+0000 7fa7d8447700 1 -- 172.21.15.115:0/2898868229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 msgr2=0x7fa7d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.744+0000 7fa7d8447700 1 --2- 172.21.15.115:0/2898868229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d00f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa7c0009fb0 tx=0x7fa7c0009b30).stop 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.744+0000 7fa7d8447700 1 -- 172.21.15.115:0/2898868229 shutdown_connections 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.744+0000 7fa7d8447700 1 --2- 172.21.15.115:0/2898868229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d00f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.744+0000 7fa7d8447700 1 -- 172.21.15.115:0/2898868229 >> 172.21.15.115:0/2898868229 conn(0x7fa7d00f0b70 msgr2=0x7fa7d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 -- 172.21.15.115:0/2898868229 shutdown_connections 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 -- 172.21.15.115:0/2898868229 wait complete. 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 Processor -- start 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 -- start start 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:23.749 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d8447700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0106bd0 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d61e3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d61e3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44424/0 (socket says 172.21.15.115:44424) 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.745+0000 7fa7d61e3700 1 -- 172.21.15.115:0/1414257920 learned_addr learned my addr 172.21.15.115:0/1414257920 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7d61e3700 1 -- 172.21.15.115:0/1414257920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0005040 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7d61e3700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa7c0013510 tx=0x7fa7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c0009320 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7c0013f00 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d0106dd0 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c0014d90 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.746+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d0107260 con 0x7fa7d00f5410 2024-01-21T07:31:23.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.747+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7c00263a0 con 0x7fa7d00f5410 2024-01-21T07:31:23.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.747+0000 7fa7c6ffd700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 0x7fa7bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:23.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.747+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7c0040ff0 con 0x7fa7d00f5410 2024-01-21T07:31:23.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.747+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d0059100 con 0x7fa7d00f5410 2024-01-21T07:31:23.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.747+0000 7fa7d59e2700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 0x7fa7bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:23.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.748+0000 7fa7d59e2700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 0x7fa7bc032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa7cc0096a0 tx=0x7fa7cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:23.754 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.752+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7c0012920 con 0x7fa7d00f5410 2024-01-21T07:31:23.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.958+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7d00fae60 con 0x7fa7d00f5410 2024-01-21T07:31:23.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.959+0000 7fa7c6ffd700 1 -- 172.21.15.115:0/1414257920 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa7c002b020 con 0x7fa7d00f5410 2024-01-21T07:31:23.961 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:23.961 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 msgr2=0x7fa7bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 0x7fa7bc032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa7cc0096a0 tx=0x7fa7cc006b40).stop 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 msgr2=0x7fa7d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa7c0013510 tx=0x7fa7c000b040).stop 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 shutdown_connections 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7bc02ff60 0x7fa7bc032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 --2- 172.21.15.115:0/1414257920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7d00f5410 0x7fa7d01066e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 >> 172.21.15.115:0/1414257920 conn(0x7fa7d00f0b70 msgr2=0x7fa7d005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 shutdown_connections 2024-01-21T07:31:23.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:23.960+0000 7fa7d8447700 1 -- 172.21.15.115:0/1414257920 wait complete. 2024-01-21T07:31:23.964 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:23.830051+0000 mgr.smithi107.adejht (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:23.960082+0000 mon.smithi107 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.115:0/1414257920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:25.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:25.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:25.830349+0000 mgr.smithi107.adejht (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- 172.21.15.115:0/4137997733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 msgr2=0x7ff7240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 --2- 172.21.15.115:0/4137997733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff7240f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7ff718009fb0 tx=0x7ff718009b30).stop 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- 172.21.15.115:0/4137997733 shutdown_connections 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 --2- 172.21.15.115:0/4137997733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff7240f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- 172.21.15.115:0/4137997733 >> 172.21.15.115:0/4137997733 conn(0x7ff7240f1c70 msgr2=0x7ff7240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:26.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- 172.21.15.115:0/4137997733 shutdown_connections 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- 172.21.15.115:0/4137997733 wait complete. 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 Processor -- start 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.446+0000 7ff72acc5700 1 -- start start 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff72acc5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff72acc5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff724106c20 con 0x7ff7240f6510 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff728a61700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff728a61700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44444/0 (socket says 172.21.15.115:44444) 2024-01-21T07:31:26.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff728a61700 1 -- 172.21.15.115:0/2091914362 learned_addr learned my addr 172.21.15.115:0/2091914362 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff728a61700 1 -- 172.21.15.115:0/2091914362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff718005040 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff728a61700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff718013510 tx=0x7ff71800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff718009320 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff718013f00 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff724106e20 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.447+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff718014d90 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.448+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7241072b0 con 0x7ff7240f6510 2024-01-21T07:31:26.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.448+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7180095f0 con 0x7ff7240f6510 2024-01-21T07:31:26.453 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.448+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff724045ad0 con 0x7ff7240f6510 2024-01-21T07:31:26.453 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.449+0000 7ff7217fa700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 0x7ff714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:26.453 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.449+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff718041000 con 0x7ff7240f6510 2024-01-21T07:31:26.453 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.449+0000 7ff723fff700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 0x7ff714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:26.453 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.449+0000 7ff723fff700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 0x7ff714032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff7100096a0 tx=0x7ff710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:26.455 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.453+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff718014420 con 0x7ff7240f6510 2024-01-21T07:31:26.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.660+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7240fbf60 con 0x7ff7240f6510 2024-01-21T07:31:26.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.661+0000 7ff7217fa700 1 -- 172.21.15.115:0/2091914362 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff71801d7f0 con 0x7ff7240f6510 2024-01-21T07:31:26.663 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:26.663 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:26.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 msgr2=0x7ff714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:26.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 0x7ff714032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff7100096a0 tx=0x7ff710006b40).stop 2024-01-21T07:31:26.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 msgr2=0x7ff724106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:26.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff718013510 tx=0x7ff71800b040).stop 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 shutdown_connections 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff71402ff60 0x7ff714032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 --2- 172.21.15.115:0/2091914362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7240f6510 0x7ff724106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 >> 172.21.15.115:0/2091914362 conn(0x7ff7240f1c70 msgr2=0x7ff7240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 shutdown_connections 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:26.662+0000 7ff72acc5700 1 -- 172.21.15.115:0/2091914362 wait complete. 2024-01-21T07:31:26.665 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:27.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:26 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:26.661952+0000 mon.smithi107 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.115:0/2091914362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:28.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:28.036 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:27.830653+0000 mgr.smithi107.adejht (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 -- 172.21.15.115:0/3431721797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 msgr2=0x7f6ebc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3431721797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc0f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6eb8009fb0 tx=0x7f6eb8009b30).stop 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 -- 172.21.15.115:0/3431721797 shutdown_connections 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3431721797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc0f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 -- 172.21.15.115:0/3431721797 >> 172.21.15.115:0/3431721797 conn(0x7f6ebc0f1c70 msgr2=0x7f6ebc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 -- 172.21.15.115:0/3431721797 shutdown_connections 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.172+0000 7f6ec381c700 1 -- 172.21.15.115:0/3431721797 wait complete. 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec381c700 1 Processor -- start 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec381c700 1 -- start start 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec381c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:29.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec381c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc069e20 con 0x7f6ebc0f6510 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec15b8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec15b8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44468/0 (socket says 172.21.15.115:44468) 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec15b8700 1 -- 172.21.15.115:0/3804962173 learned_addr learned my addr 172.21.15.115:0/3804962173 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec15b8700 1 -- 172.21.15.115:0/3804962173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8005040 con 0x7f6ebc0f6510 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.173+0000 7f6ec15b8700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6eb8013510 tx=0x7f6eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.174+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eb8009320 con 0x7f6ebc0f6510 2024-01-21T07:31:29.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.174+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc069fc0 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.174+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6eb8013f00 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.174+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eb8014d90 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.174+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc06a450 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6eb8014420 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc059100 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6eb1ffb700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 0x7f6ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6eb8041110 con 0x7f6ebc0f6510 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6ec0db7700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 0x7f6ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:29.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.175+0000 7f6ec0db7700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 0x7f6ea8032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6eac0096a0 tx=0x7f6eac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:29.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.180+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6eb801db70 con 0x7f6ebc0f6510 2024-01-21T07:31:29.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.386+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ebc045ad0 con 0x7f6ebc0f6510 2024-01-21T07:31:29.389 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.387+0000 7f6eb1ffb700 1 -- 172.21.15.115:0/3804962173 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6eb80242b0 con 0x7f6ebc0f6510 2024-01-21T07:31:29.389 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:29.389 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 msgr2=0x7f6ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 0x7f6ea8032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6eac0096a0 tx=0x7f6eac006b40).stop 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 msgr2=0x7f6ebc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6eb8013510 tx=0x7f6eb800b040).stop 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 shutdown_connections 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6ea802ff60 0x7f6ea8032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 --2- 172.21.15.115:0/3804962173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ebc0f6510 0x7f6ebc06b9a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 >> 172.21.15.115:0/3804962173 conn(0x7f6ebc0f1c70 msgr2=0x7f6ebc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.388+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 shutdown_connections 2024-01-21T07:31:29.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:29.389+0000 7f6ec381c700 1 -- 172.21.15.115:0/3804962173 wait complete. 2024-01-21T07:31:29.392 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:29.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:29.388251+0000 mon.smithi107 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.115:0/3804962173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:30.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:30.772 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:30.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:29.831041+0000 mgr.smithi107.adejht (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:31.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.879+0000 7fd86b35b700 1 -- 172.21.15.115:0/885984055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 msgr2=0x7fd8640f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:31.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.879+0000 7fd86b35b700 1 --2- 172.21.15.115:0/885984055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8640f66c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd854009fb0 tx=0x7fd854009b30).stop 2024-01-21T07:31:31.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.879+0000 7fd86b35b700 1 -- 172.21.15.115:0/885984055 shutdown_connections 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.879+0000 7fd86b35b700 1 --2- 172.21.15.115:0/885984055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8640f66c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.879+0000 7fd86b35b700 1 -- 172.21.15.115:0/885984055 >> 172.21.15.115:0/885984055 conn(0x7fd8640f1aa0 msgr2=0x7fd8640f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 -- 172.21.15.115:0/885984055 shutdown_connections 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 -- 172.21.15.115:0/885984055 wait complete. 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 Processor -- start 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 -- start start 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86b35b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8641069a0 con 0x7fd8640f62a0 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86a359700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86a359700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60396/0 (socket says 172.21.15.115:60396) 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.880+0000 7fd86a359700 1 -- 172.21.15.115:0/98588880 learned_addr learned my addr 172.21.15.115:0/98588880 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd86a359700 1 -- 172.21.15.115:0/98588880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd854005040 con 0x7fd8640f62a0 2024-01-21T07:31:31.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd86a359700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd854013510 tx=0x7fd85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:31.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd854009320 con 0x7fd8640f62a0 2024-01-21T07:31:31.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd854013f00 con 0x7fd8640f62a0 2024-01-21T07:31:31.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd864106ba0 con 0x7fd8640f62a0 2024-01-21T07:31:31.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd854014dc0 con 0x7fd8640f62a0 2024-01-21T07:31:31.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.881+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864107030 con 0x7fd8640f62a0 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.882+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8540095f0 con 0x7fd8640f62a0 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.882+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd864045ad0 con 0x7fd8640f62a0 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.882+0000 7fd85affd700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 0x7fd850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.882+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd854041000 con 0x7fd8640f62a0 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.882+0000 7fd869b58700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 0x7fd850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:31.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.883+0000 7fd869b58700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 0x7fd850032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:31.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:31.887+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd85406d2b0 con 0x7fd8640f62a0 2024-01-21T07:31:32.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.094+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd864107ee0 con 0x7fd8640f62a0 2024-01-21T07:31:32.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.095+0000 7fd85affd700 1 -- 172.21.15.115:0/98588880 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd854010230 con 0x7fd8640f62a0 2024-01-21T07:31:32.098 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:32.098 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:32.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 msgr2=0x7fd850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:32.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 0x7fd850032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).stop 2024-01-21T07:31:32.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 msgr2=0x7fd8641064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:32.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd854013510 tx=0x7fd85400b040).stop 2024-01-21T07:31:32.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 shutdown_connections 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd85002ff60 0x7fd850032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 --2- 172.21.15.115:0/98588880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8640f62a0 0x7fd8641064b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 >> 172.21.15.115:0/98588880 conn(0x7fd8640f1aa0 msgr2=0x7fd8640f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 shutdown_connections 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:32.097+0000 7fd86b35b700 1 -- 172.21.15.115:0/98588880 wait complete. 2024-01-21T07:31:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:31.831405+0000 mgr.smithi107.adejht (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:32.096614+0000 mon.smithi107 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.115:0/98588880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:33.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:33.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:33.831801+0000 mgr.smithi107.adejht (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:34.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 -- 172.21.15.115:0/2543068597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 msgr2=0x7f2a500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:34.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2543068597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a500f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f2a40009fb0 tx=0x7f2a40009b30).stop 2024-01-21T07:31:34.636 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 -- 172.21.15.115:0/2543068597 shutdown_connections 2024-01-21T07:31:34.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2543068597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a500f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:34.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 -- 172.21.15.115:0/2543068597 >> 172.21.15.115:0/2543068597 conn(0x7f2a500f0b70 msgr2=0x7f2a500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:34.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 -- 172.21.15.115:0/2543068597 shutdown_connections 2024-01-21T07:31:34.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.633+0000 7f2a5869c700 1 -- 172.21.15.115:0/2543068597 wait complete. 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a5869c700 1 Processor -- start 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a5869c700 1 -- start start 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a5869c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a5869c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a50060220 con 0x7f2a500f5410 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a56438700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a56438700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60414/0 (socket says 172.21.15.115:60414) 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a56438700 1 -- 172.21.15.115:0/2815391556 learned_addr learned my addr 172.21.15.115:0/2815391556 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:34.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.634+0000 7f2a56438700 1 -- 172.21.15.115:0/2815391556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a40005040 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a56438700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2a40013510 tx=0x7f2a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a40009320 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a40013f00 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a50060420 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a40014d90 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.635+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a50109b00 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a400095f0 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a46ffd700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 0x7f2a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a50059100 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a400410a0 con 0x7f2a500f5410 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a55c37700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 0x7f2a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:34.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.636+0000 7f2a55c37700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 0x7f2a3c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2a4c0096a0 tx=0x7f2a4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:34.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.641+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a4001d7f0 con 0x7f2a500f5410 2024-01-21T07:31:34.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.850+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a50045ad0 con 0x7f2a500f5410 2024-01-21T07:31:34.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.850+0000 7f2a46ffd700 1 -- 172.21.15.115:0/2815391556 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2a4002c090 con 0x7f2a500f5410 2024-01-21T07:31:34.853 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:34.853 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 msgr2=0x7f2a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 0x7f2a3c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2a4c0096a0 tx=0x7f2a4c006b40).stop 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 msgr2=0x7f2a50109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2a40013510 tx=0x7f2a4000b040).stop 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 shutdown_connections 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a3c02ff60 0x7f2a3c032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:34.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 --2- 172.21.15.115:0/2815391556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a500f5410 0x7f2a50109450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:34.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 >> 172.21.15.115:0/2815391556 conn(0x7f2a500f0b70 msgr2=0x7f2a500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:34.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 shutdown_connections 2024-01-21T07:31:34.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:34.852+0000 7f2a5869c700 1 -- 172.21.15.115:0/2815391556 wait complete. 2024-01-21T07:31:34.855 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:35.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:34.852001+0000 mon.smithi107 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.115:0/2815391556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:36.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:36.237 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:35.832112+0000 mgr.smithi107.adejht (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 -- 172.21.15.115:0/7827690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 msgr2=0x7f0ee40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/7827690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee40f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0ed8009fb0 tx=0x7f0ed8009b30).stop 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 -- 172.21.15.115:0/7827690 shutdown_connections 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/7827690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee40f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 -- 172.21.15.115:0/7827690 >> 172.21.15.115:0/7827690 conn(0x7f0ee40f1c70 msgr2=0x7f0ee40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 -- 172.21.15.115:0/7827690 shutdown_connections 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.358+0000 7f0ee8a34700 1 -- 172.21.15.115:0/7827690 wait complete. 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee8a34700 1 Processor -- start 2024-01-21T07:31:37.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee8a34700 1 -- start start 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee8a34700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee8a34700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4106c20 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60436/0 (socket says 172.21.15.115:60436) 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee259c700 1 -- 172.21.15.115:0/3106262575 learned_addr learned my addr 172.21.15.115:0/3106262575 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.359+0000 7f0ee259c700 1 -- 172.21.15.115:0/3106262575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8005040 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ee259c700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0ed8013510 tx=0x7f0ed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ed8009320 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ed8013f00 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee4106e20 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ed8014d90 con 0x7f0ee40f6510 2024-01-21T07:31:37.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee41072b0 con 0x7f0ee40f6510 2024-01-21T07:31:37.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.360+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ed8014420 con 0x7f0ee40f6510 2024-01-21T07:31:37.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.361+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee4045ad0 con 0x7f0ee40f6510 2024-01-21T07:31:37.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.361+0000 7f0ed2ffd700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 0x7f0ecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:37.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.361+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ed80410b0 con 0x7f0ee40f6510 2024-01-21T07:31:37.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.361+0000 7f0ee1d9b700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 0x7f0ecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:37.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.361+0000 7f0ee1d9b700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 0x7f0ecc032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0ed40096a0 tx=0x7f0ed4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:37.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.366+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ed80095f0 con 0x7f0ee40f6510 2024-01-21T07:31:37.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.572+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ee40fbf60 con 0x7f0ee40f6510 2024-01-21T07:31:37.575 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.573+0000 7f0ed2ffd700 1 -- 172.21.15.115:0/3106262575 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ed801db70 con 0x7f0ee40f6510 2024-01-21T07:31:37.576 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:37.576 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 msgr2=0x7f0ecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 0x7f0ecc032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0ed40096a0 tx=0x7f0ed4006b40).stop 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 msgr2=0x7f0ee4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0ed8013510 tx=0x7f0ed800b040).stop 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 shutdown_connections 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0ecc02ff60 0x7f0ecc032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 --2- 172.21.15.115:0/3106262575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ee40f6510 0x7f0ee4106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 >> 172.21.15.115:0/3106262575 conn(0x7f0ee40f1c70 msgr2=0x7f0ee40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 shutdown_connections 2024-01-21T07:31:37.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:37.574+0000 7f0ee8a34700 1 -- 172.21.15.115:0/3106262575 wait complete. 2024-01-21T07:31:37.578 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:38.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:37.574209+0000 mon.smithi107 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.115:0/3106262575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:38.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:38.983 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:39.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:37.832445+0000 mgr.smithi107.adejht (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:40.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 -- 172.21.15.115:0/782333163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 msgr2=0x7fedc80f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 --2- 172.21.15.115:0/782333163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc80f4fa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fedb8009fb0 tx=0x7fedb8009b30).stop 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 -- 172.21.15.115:0/782333163 shutdown_connections 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 --2- 172.21.15.115:0/782333163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc80f4fa0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 -- 172.21.15.115:0/782333163 >> 172.21.15.115:0/782333163 conn(0x7fedc80f02e0 msgr2=0x7fedc80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 -- 172.21.15.115:0/782333163 shutdown_connections 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.098+0000 7fedcf507700 1 -- 172.21.15.115:0/782333163 wait complete. 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.099+0000 7fedcf507700 1 Processor -- start 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.099+0000 7fedcf507700 1 -- start start 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.099+0000 7fedcf507700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.099+0000 7fedcf507700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8105ae0 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcd2a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcd2a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36824/0 (socket says 172.21.15.115:36824) 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcd2a3700 1 -- 172.21.15.115:0/389261055 learned_addr learned my addr 172.21.15.115:0/389261055 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcd2a3700 1 -- 172.21.15.115:0/389261055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8005040 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcd2a3700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fedb8013510 tx=0x7fedb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedb8009320 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc8105ce0 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedb8013f00 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedb8014d90 con 0x7fedc80f4b80 2024-01-21T07:31:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.100+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc8106170 con 0x7fedc80f4b80 2024-01-21T07:31:40.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.101+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc8045ad0 con 0x7fedc80f4b80 2024-01-21T07:31:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.101+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedb80095f0 con 0x7fedc80f4b80 2024-01-21T07:31:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.101+0000 7fedbdffb700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 0x7fedb4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.101+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedb8040f90 con 0x7fedc80f4b80 2024-01-21T07:31:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.102+0000 7fedccaa2700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 0x7fedb4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.102+0000 7fedccaa2700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 0x7fedb4032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fedc40096a0 tx=0x7fedc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:40.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.107+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedb8014420 con 0x7fedc80f4b80 2024-01-21T07:31:40.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.311+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedc80fa5d0 con 0x7fedc80f4b80 2024-01-21T07:31:40.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.311+0000 7fedbdffb700 1 -- 172.21.15.115:0/389261055 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fedb801d7f0 con 0x7fedc80f4b80 2024-01-21T07:31:40.313 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:40.313 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:40.315 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 msgr2=0x7fedb4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:40.315 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 0x7fedb4032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fedc40096a0 tx=0x7fedc4006b40).stop 2024-01-21T07:31:40.315 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 msgr2=0x7fedc81055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fedb8013510 tx=0x7fedb800b040).stop 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 shutdown_connections 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedb402ffb0 0x7fedb4032470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 --2- 172.21.15.115:0/389261055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedc80f4b80 0x7fedc81055f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 >> 172.21.15.115:0/389261055 conn(0x7fedc80f02e0 msgr2=0x7fedc80f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 shutdown_connections 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:40.313+0000 7fedcf507700 1 -- 172.21.15.115:0/389261055 wait complete. 2024-01-21T07:31:40.316 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:40.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:39.832811+0000 mgr.smithi107.adejht (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:40.312676+0000 mon.smithi107 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.115:0/389261055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:41.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:41.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 -- 172.21.15.115:0/4284206035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 msgr2=0x7efd040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 --2- 172.21.15.115:0/4284206035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd040f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7efcf4009fb0 tx=0x7efcf4009b30).stop 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 -- 172.21.15.115:0/4284206035 shutdown_connections 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 --2- 172.21.15.115:0/4284206035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd040f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 -- 172.21.15.115:0/4284206035 >> 172.21.15.115:0/4284206035 conn(0x7efd040f09a0 msgr2=0x7efd040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 -- 172.21.15.115:0/4284206035 shutdown_connections 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.851+0000 7efd0b53c700 1 -- 172.21.15.115:0/4284206035 wait complete. 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.852+0000 7efd0b53c700 1 Processor -- start 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.852+0000 7efd0b53c700 1 -- start start 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.852+0000 7efd0b53c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.852+0000 7efd0b53c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd041058a0 con 0x7efd040f51a0 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0a53a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0a53a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36850/0 (socket says 172.21.15.115:36850) 2024-01-21T07:31:42.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0a53a700 1 -- 172.21.15.115:0/293869283 learned_addr learned my addr 172.21.15.115:0/293869283 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0a53a700 1 -- 172.21.15.115:0/293869283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf4005040 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0a53a700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7efcf4013510 tx=0x7efcf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcf4009320 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd04105aa0 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcf4013f00 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcf4014dc0 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.853+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd04105f30 con 0x7efd040f51a0 2024-01-21T07:31:42.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.854+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efcf40095f0 con 0x7efd040f51a0 2024-01-21T07:31:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.854+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd04045ad0 con 0x7efd040f51a0 2024-01-21T07:31:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.854+0000 7efcfaffd700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 0x7efcf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.854+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efcf4042dd0 con 0x7efd040f51a0 2024-01-21T07:31:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.855+0000 7efd09d39700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 0x7efcf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:42.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.855+0000 7efd09d39700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 0x7efcf0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7efd000096a0 tx=0x7efd00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:42.860 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:42.859+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efcf4014420 con 0x7efd040f51a0 2024-01-21T07:31:43.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:41.833169+0000 mgr.smithi107.adejht (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:43.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.065+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd04106d80 con 0x7efd040f51a0 2024-01-21T07:31:43.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.065+0000 7efcfaffd700 1 -- 172.21.15.115:0/293869283 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efcf4011290 con 0x7efd040f51a0 2024-01-21T07:31:43.066 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:43.067 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 msgr2=0x7efcf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 0x7efcf0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7efd000096a0 tx=0x7efd00006b40).stop 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 msgr2=0x7efd041053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7efcf4013510 tx=0x7efcf400b040).stop 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 shutdown_connections 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efcf002ff60 0x7efcf0032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 --2- 172.21.15.115:0/293869283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd040f51a0 0x7efd041053b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.066+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 >> 172.21.15.115:0/293869283 conn(0x7efd040f09a0 msgr2=0x7efd040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.067+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 shutdown_connections 2024-01-21T07:31:43.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:43.067+0000 7efd0b53c700 1 -- 172.21.15.115:0/293869283 wait complete. 2024-01-21T07:31:43.069 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:44.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:43.065522+0000 mon.smithi107 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.115:0/293869283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:44.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:44.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:43.833447+0000 mgr.smithi107.adejht (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/1022547295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 msgr2=0x7fcfc40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/1022547295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f9150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fcfb4009fb0 tx=0x7fcfb4009b30).stop 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/1022547295 shutdown_connections 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/1022547295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f9150 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/1022547295 >> 172.21.15.115:0/1022547295 conn(0x7fcfc40f0b70 msgr2=0x7fcfc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/1022547295 shutdown_connections 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.610+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/1022547295 wait complete. 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc9f3e700 1 Processor -- start 2024-01-21T07:31:45.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc9f3e700 1 -- start start 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc9f3e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc9f3e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40f5050 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36868/0 (socket says 172.21.15.115:36868) 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.611+0000 7fcfc37fe700 1 -- 172.21.15.115:0/3837390175 learned_addr learned my addr 172.21.15.115:0/3837390175 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfc37fe700 1 -- 172.21.15.115:0/3837390175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4005040 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfc37fe700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcfb4013510 tx=0x7fcfb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb4009320 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc40f51f0 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfb4013f00 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb4014dc0 con 0x7fcfc4060220 2024-01-21T07:31:45.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.612+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc40f5680 con 0x7fcfc4060220 2024-01-21T07:31:45.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.613+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfb4014420 con 0x7fcfc4060220 2024-01-21T07:31:45.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.613+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc4059100 con 0x7fcfc4060220 2024-01-21T07:31:45.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.613+0000 7fcfbbfff700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 0x7fcfb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:45.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.613+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfb4041020 con 0x7fcfc4060220 2024-01-21T07:31:45.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.613+0000 7fcfc2ffd700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 0x7fcfb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:45.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.614+0000 7fcfc2ffd700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 0x7fcfb0032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fcfac0096a0 tx=0x7fcfac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:45.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.618+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfb40095f0 con 0x7fcfc4060220 2024-01-21T07:31:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.824+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfc40fbf50 con 0x7fcfc4060220 2024-01-21T07:31:45.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.825+0000 7fcfbbfff700 1 -- 172.21.15.115:0/3837390175 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcfb401cb10 con 0x7fcfc4060220 2024-01-21T07:31:45.827 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:45.827 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 msgr2=0x7fcfb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:45.828 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 0x7fcfb0032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fcfac0096a0 tx=0x7fcfac006b40).stop 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 msgr2=0x7fcfc40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcfb4013510 tx=0x7fcfb400b040).stop 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 shutdown_connections 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcfb002ff60 0x7fcfb0032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 --2- 172.21.15.115:0/3837390175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfc4060220 0x7fcfc40f6bd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 >> 172.21.15.115:0/3837390175 conn(0x7fcfc40f0b70 msgr2=0x7fcfc40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 shutdown_connections 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:45.827+0000 7fcfc9f3e700 1 -- 172.21.15.115:0/3837390175 wait complete. 2024-01-21T07:31:45.829 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:45.825725+0000 mon.smithi107 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.115:0/3837390175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:47.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:47.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:45.833741+0000 mgr.smithi107.adejht (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:48.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 -- 172.21.15.115:0/423236330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 msgr2=0x7f34600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 --2- 172.21.15.115:0/423236330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f34600f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3448009fb0 tx=0x7f3448009b30).stop 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 -- 172.21.15.115:0/423236330 shutdown_connections 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 --2- 172.21.15.115:0/423236330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f34600f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 -- 172.21.15.115:0/423236330 >> 172.21.15.115:0/423236330 conn(0x7f34600f0b70 msgr2=0x7f34600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 -- 172.21.15.115:0/423236330 shutdown_connections 2024-01-21T07:31:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.339+0000 7f34653b9700 1 -- 172.21.15.115:0/423236330 wait complete. 2024-01-21T07:31:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f34653b9700 1 Processor -- start 2024-01-21T07:31:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f34653b9700 1 -- start start 2024-01-21T07:31:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f34653b9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f34653b9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460060280 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f345effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f345effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36890/0 (socket says 172.21.15.115:36890) 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.340+0000 7f345effd700 1 -- 172.21.15.115:0/2383624761 learned_addr learned my addr 172.21.15.115:0/2383624761 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f345effd700 1 -- 172.21.15.115:0/2383624761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448005040 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f345effd700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3448013510 tx=0x7f344800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3448009320 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3448013f00 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3460060480 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3448014d90 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.341+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460109ac0 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.342+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34480095f0 con 0x7f34600f5410 2024-01-21T07:31:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.342+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3460045ad0 con 0x7f34600f5410 2024-01-21T07:31:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.342+0000 7f34577fe700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 0x7f344c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.342+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34480410c0 con 0x7f34600f5410 2024-01-21T07:31:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.342+0000 7f345e7fc700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 0x7f344c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.343+0000 7f345e7fc700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 0x7f344c032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f34500096a0 tx=0x7f3450006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:48.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.347+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3448014420 con 0x7f34600f5410 2024-01-21T07:31:48.556 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.555+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34600fae60 con 0x7f34600f5410 2024-01-21T07:31:48.557 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.556+0000 7f34577fe700 1 -- 172.21.15.115:0/2383624761 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3448010230 con 0x7f34600f5410 2024-01-21T07:31:48.557 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:48.557 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 msgr2=0x7f344c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 0x7f344c032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f34500096a0 tx=0x7f3450006b40).stop 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 msgr2=0x7f3460109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3448013510 tx=0x7f344800b040).stop 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 shutdown_connections 2024-01-21T07:31:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f344c02ff60 0x7f344c032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 --2- 172.21.15.115:0/2383624761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34600f5410 0x7f3460109390 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 >> 172.21.15.115:0/2383624761 conn(0x7f34600f0b70 msgr2=0x7f34600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 shutdown_connections 2024-01-21T07:31:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:48.558+0000 7f34653b9700 1 -- 172.21.15.115:0/2383624761 wait complete. 2024-01-21T07:31:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:47.834109+0000 mgr.smithi107.adejht (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:48.556194+0000 mon.smithi107 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.115:0/2383624761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:49.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:49.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:51.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- 172.21.15.115:0/2541909746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 msgr2=0x7f05900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:51.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2541909746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f05900f55c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0584009fb0 tx=0x7f0584009b30).stop 2024-01-21T07:31:51.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- 172.21.15.115:0/2541909746 shutdown_connections 2024-01-21T07:31:51.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2541909746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f05900f55c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:51.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- 172.21.15.115:0/2541909746 >> 172.21.15.115:0/2541909746 conn(0x7f05900f09a0 msgr2=0x7f05900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- 172.21.15.115:0/2541909746 shutdown_connections 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- 172.21.15.115:0/2541909746 wait complete. 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 Processor -- start 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.093+0000 7f0595ad7700 1 -- start start 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0595ad7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0595ad7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590106fc0 con 0x7f05900f51a0 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0594ad5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0594ad5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44742/0 (socket says 172.21.15.115:44742) 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0594ad5700 1 -- 172.21.15.115:0/2278419715 learned_addr learned my addr 172.21.15.115:0/2278419715 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0594ad5700 1 -- 172.21.15.115:0/2278419715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0584005040 con 0x7f05900f51a0 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.094+0000 7f0594ad5700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0584009830 tx=0x7f058400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:51.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0584009320 con 0x7f05900f51a0 2024-01-21T07:31:51.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0590060220 con 0x7f05900f51a0 2024-01-21T07:31:51.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05840136c0 con 0x7f05900f51a0 2024-01-21T07:31:51.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f058401ee90 con 0x7f05900f51a0 2024-01-21T07:31:51.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059010bac0 con 0x7f05900f51a0 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.095+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f058401e420 con 0x7f05900f51a0 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.096+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0590045ad0 con 0x7f05900f51a0 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.096+0000 7f058d7fa700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 0x7f05800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.096+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0584040ff0 con 0x7f05900f51a0 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.096+0000 7f058ffff700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 0x7f05800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:51.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.096+0000 7f058ffff700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 0x7f05800323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f057c0096a0 tx=0x7f057c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:51.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.100+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05840095f0 con 0x7f05900f51a0 2024-01-21T07:31:51.308 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.307+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f059010c280 con 0x7f05900f51a0 2024-01-21T07:31:51.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.308+0000 7f058d7fa700 1 -- 172.21.15.115:0/2278419715 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f058401cb10 con 0x7f05900f51a0 2024-01-21T07:31:51.309 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:51.309 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:51.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 msgr2=0x7f05800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:51.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 0x7f05800323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f057c0096a0 tx=0x7f057c006b40).stop 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 msgr2=0x7f059010b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0584009830 tx=0x7f058400b040).stop 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 shutdown_connections 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f058002ff10 0x7f05800323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 --2- 172.21.15.115:0/2278419715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05900f51a0 0x7f059010b5d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 >> 172.21.15.115:0/2278419715 conn(0x7f05900f09a0 msgr2=0x7f05900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 shutdown_connections 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:51.309+0000 7f0595ad7700 1 -- 172.21.15.115:0/2278419715 wait complete. 2024-01-21T07:31:51.311 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:51.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:49.834461+0000 mgr.smithi107.adejht (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:52.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:51.308184+0000 mon.smithi107 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.115:0/2278419715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:52.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:52.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:53.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:51.834772+0000 mgr.smithi107.adejht (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 -- 172.21.15.115:0/2878585523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 msgr2=0x7feddc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 --2- 172.21.15.115:0/2878585523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc0f90f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fedc4009fb0 tx=0x7fedc4009b30).stop 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 -- 172.21.15.115:0/2878585523 shutdown_connections 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 --2- 172.21.15.115:0/2878585523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc0f90f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 -- 172.21.15.115:0/2878585523 >> 172.21.15.115:0/2878585523 conn(0x7feddc0f0b70 msgr2=0x7feddc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 -- 172.21.15.115:0/2878585523 shutdown_connections 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 -- 172.21.15.115:0/2878585523 wait complete. 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.812+0000 7fede292b700 1 Processor -- start 2024-01-21T07:31:53.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7fede292b700 1 -- start start 2024-01-21T07:31:53.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7fede292b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:53.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7fede292b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc106bd0 con 0x7feddc060280 2024-01-21T07:31:53.816 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7feddbfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:53.816 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7feddbfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44766/0 (socket says 172.21.15.115:44766) 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7feddbfff700 1 -- 172.21.15.115:0/2659214832 learned_addr learned my addr 172.21.15.115:0/2659214832 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.813+0000 7feddbfff700 1 -- 172.21.15.115:0/2659214832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc4005040 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7feddbfff700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fedc4013510 tx=0x7fedc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc4009320 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc106dd0 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedc4013f00 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc4014dc0 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.814+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc107260 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedc40095f0 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7fedd8ff9700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 0x7fedc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedc4041030 con 0x7feddc060280 2024-01-21T07:31:53.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7feddb7fe700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 0x7fedc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:53.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc059100 con 0x7feddc060280 2024-01-21T07:31:53.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.815+0000 7feddb7fe700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 0x7fedc8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:53.821 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:53.820+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedc4014420 con 0x7feddc060280 2024-01-21T07:31:54.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.025+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feddc045ad0 con 0x7feddc060280 2024-01-21T07:31:54.027 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.026+0000 7fedd8ff9700 1 -- 172.21.15.115:0/2659214832 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fedc402b020 con 0x7feddc060280 2024-01-21T07:31:54.027 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:54.027 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:54.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 msgr2=0x7fedc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:54.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 0x7fedc8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).stop 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 msgr2=0x7feddc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fedc4013510 tx=0x7fedc400b040).stop 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 shutdown_connections 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fedc802ff60 0x7fedc8032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 --2- 172.21.15.115:0/2659214832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feddc060280 0x7feddc1066e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 >> 172.21.15.115:0/2659214832 conn(0x7feddc0f0b70 msgr2=0x7feddc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 shutdown_connections 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:54.027+0000 7fede292b700 1 -- 172.21.15.115:0/2659214832 wait complete. 2024-01-21T07:31:54.029 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:53.835031+0000 mgr.smithi107.adejht (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:54.026162+0000 mon.smithi107 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.115:0/2659214832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:55.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:55.422 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:55.835361+0000 mgr.smithi107.adejht (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:56.601 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.597+0000 7f5bcd76c700 1 -- 172.21.15.115:0/2862139740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 msgr2=0x7f5bc80f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.597+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/2862139740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc80f8f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5bb8009fb0 tx=0x7f5bb8009b30).stop 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- 172.21.15.115:0/2862139740 shutdown_connections 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/2862139740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc80f8f80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- 172.21.15.115:0/2862139740 >> 172.21.15.115:0/2862139740 conn(0x7f5bc80f09a0 msgr2=0x7f5bc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- 172.21.15.115:0/2862139740 shutdown_connections 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- 172.21.15.115:0/2862139740 wait complete. 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 Processor -- start 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- start start 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.598+0000 7f5bcd76c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc8106a20 con 0x7f5bc8060180 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:44778/0 (socket says 172.21.15.115:44778) 2024-01-21T07:31:56.602 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc7fff700 1 -- 172.21.15.115:0/735824041 learned_addr learned my addr 172.21.15.115:0/735824041 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc7fff700 1 -- 172.21.15.115:0/735824041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8005040 con 0x7f5bc8060180 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc7fff700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5bb8013510 tx=0x7f5bb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb8009320 con 0x7f5bc8060180 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bb8013f00 con 0x7f5bc8060180 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc8106c20 con 0x7f5bc8060180 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb8014d90 con 0x7f5bc8060180 2024-01-21T07:31:56.603 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.599+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc81070b0 con 0x7f5bc8060180 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.600+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bb80263a0 con 0x7f5bc8060180 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.600+0000 7f5bc4ff9700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 0x7f5bb0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.600+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bb80410a0 con 0x7f5bc8060180 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.600+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc8045ad0 con 0x7f5bc8060180 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.600+0000 7f5bc77fe700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 0x7f5bb0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.601+0000 7f5bc77fe700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 0x7f5bb0032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:56.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.605+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bb8012920 con 0x7f5bc8060180 2024-01-21T07:31:56.811 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.809+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bc8107dd0 con 0x7f5bc8060180 2024-01-21T07:31:56.812 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.810+0000 7f5bc4ff9700 1 -- 172.21.15.115:0/735824041 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5bb802b020 con 0x7f5bc8060180 2024-01-21T07:31:56.812 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:56.812 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:56.813 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 msgr2=0x7f5bb0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 0x7f5bb0032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).stop 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 msgr2=0x7f5bc8106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5bb8013510 tx=0x7f5bb800b040).stop 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 shutdown_connections 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5bb002ffb0 0x7f5bb0032470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 --2- 172.21.15.115:0/735824041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bc8060180 0x7f5bc8106530 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 >> 172.21.15.115:0/735824041 conn(0x7f5bc80f09a0 msgr2=0x7f5bc80f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 shutdown_connections 2024-01-21T07:31:56.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:56.811+0000 7f5bcd76c700 1 -- 172.21.15.115:0/735824041 wait complete. 2024-01-21T07:31:56.815 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:31:58.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:56.811047+0000 mon.smithi107 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.115:0/735824041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:31:58.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:31:58.226 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:31:59.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:57.835675+0000 mgr.smithi107.adejht (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:31:59.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 -- 172.21.15.115:0/4077975001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 msgr2=0x7f22980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:59.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 --2- 172.21.15.115:0/4077975001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f22980f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2288009fb0 tx=0x7f2288009b30).stop 2024-01-21T07:31:59.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 -- 172.21.15.115:0/4077975001 shutdown_connections 2024-01-21T07:31:59.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 --2- 172.21.15.115:0/4077975001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f22980f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 -- 172.21.15.115:0/4077975001 >> 172.21.15.115:0/4077975001 conn(0x7f22980f0b70 msgr2=0x7f22980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 -- 172.21.15.115:0/4077975001 shutdown_connections 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.361+0000 7f229f7d2700 1 -- 172.21.15.115:0/4077975001 wait complete. 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229f7d2700 1 Processor -- start 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229f7d2700 1 -- start start 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229f7d2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229f7d2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298106c20 con 0x7f22980f5410 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229d56e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229d56e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54338/0 (socket says 172.21.15.115:54338) 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.362+0000 7f229d56e700 1 -- 172.21.15.115:0/2576585490 learned_addr learned my addr 172.21.15.115:0/2576585490 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f229d56e700 1 -- 172.21.15.115:0/2576585490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288005040 con 0x7f22980f5410 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f229d56e700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2288013510 tx=0x7f228800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2288009320 con 0x7f22980f5410 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2288013f00 con 0x7f22980f5410 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2288014d90 con 0x7f22980f5410 2024-01-21T07:31:59.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2298106e20 con 0x7f22980f5410 2024-01-21T07:31:59.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.363+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22981072b0 con 0x7f22980f5410 2024-01-21T07:31:59.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.364+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22880095f0 con 0x7f22980f5410 2024-01-21T07:31:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.364+0000 7f228dffb700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 0x7f2284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:31:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.364+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2298045ad0 con 0x7f22980f5410 2024-01-21T07:31:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.364+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2288041000 con 0x7f22980f5410 2024-01-21T07:31:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.364+0000 7f229cd6d700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 0x7f2284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:31:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.365+0000 7f229cd6d700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 0x7f2284032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f22940096a0 tx=0x7f2294006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:31:59.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.369+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f228806d2b0 con 0x7f22980f5410 2024-01-21T07:31:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.575+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22980faf50 con 0x7f22980f5410 2024-01-21T07:31:59.578 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.576+0000 7f228dffb700 1 -- 172.21.15.115:0/2576585490 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f228801d9d0 con 0x7f22980f5410 2024-01-21T07:31:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:31:59.578 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 msgr2=0x7f2284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 0x7f2284032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f22940096a0 tx=0x7f2294006b40).stop 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 msgr2=0x7f2298106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2288013510 tx=0x7f228800b040).stop 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 shutdown_connections 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f228402ff60 0x7f2284032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 --2- 172.21.15.115:0/2576585490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22980f5410 0x7f2298106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.577+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 >> 172.21.15.115:0/2576585490 conn(0x7f22980f0b70 msgr2=0x7f22980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.578+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 shutdown_connections 2024-01-21T07:31:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:31:59.578+0000 7f229f7d2700 1 -- 172.21.15.115:0/2576585490 wait complete. 2024-01-21T07:31:59.581 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:31:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:31:59.577174+0000 mon.smithi107 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.115:0/2576585490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:00.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:00.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:31:59.836026+0000 mgr.smithi107.adejht (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 -- 172.21.15.115:0/1141164062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 msgr2=0x7f45240f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 --2- 172.21.15.115:0/1141164062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f9190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4518009fb0 tx=0x7f4518009b30).stop 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 -- 172.21.15.115:0/1141164062 shutdown_connections 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 --2- 172.21.15.115:0/1141164062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f9190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 -- 172.21.15.115:0/1141164062 >> 172.21.15.115:0/1141164062 conn(0x7f45240f0b70 msgr2=0x7f45240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 -- 172.21.15.115:0/1141164062 shutdown_connections 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.114+0000 7f4529352700 1 -- 172.21.15.115:0/1141164062 wait complete. 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4529352700 1 Processor -- start 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4529352700 1 -- start start 2024-01-21T07:32:02.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4529352700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:02.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4529352700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45240f5090 con 0x7f45240601e0 2024-01-21T07:32:02.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4522ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:02.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4522ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54360/0 (socket says 172.21.15.115:54360) 2024-01-21T07:32:02.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4522ffd700 1 -- 172.21.15.115:0/1460836324 learned_addr learned my addr 172.21.15.115:0/1460836324 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:02.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.115+0000 7f4522ffd700 1 -- 172.21.15.115:0/1460836324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4518005040 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f4522ffd700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4518004dd0 tx=0x7f4518009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4518018070 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4518013410 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45240f5230 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f451801c3c0 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45240f56c0 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.116+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f451801d420 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.117+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524059100 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.117+0000 7f450b7fe700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 0x7f450c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.117+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45180413d0 con 0x7f45240601e0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.117+0000 7f45227fc700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 0x7f450c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:02.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.117+0000 7f45227fc700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 0x7f450c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f45140096a0 tx=0x7f4514006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:02.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.121+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f451801d820 con 0x7f45240601e0 2024-01-21T07:32:02.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.329+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4524045ad0 con 0x7f45240601e0 2024-01-21T07:32:02.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.330+0000 7f450b7fe700 1 -- 172.21.15.115:0/1460836324 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f451801b370 con 0x7f45240601e0 2024-01-21T07:32:02.332 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:02.332 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:02.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 msgr2=0x7f450c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:02.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 0x7f450c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f45140096a0 tx=0x7f4514006b40).stop 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 msgr2=0x7f45240f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4518004dd0 tx=0x7f4518009b30).stop 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 shutdown_connections 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f450c02ff60 0x7f450c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 --2- 172.21.15.115:0/1460836324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45240601e0 0x7f45240f6c10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 >> 172.21.15.115:0/1460836324 conn(0x7f45240f0b70 msgr2=0x7f45240f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 shutdown_connections 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:02.331+0000 7f4529352700 1 -- 172.21.15.115:0/1460836324 wait complete. 2024-01-21T07:32:02.334 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:01.836353+0000 mgr.smithi107.adejht (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:02.330998+0000 mon.smithi107 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.115:0/1460836324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:03.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:03.735 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:03.836721+0000 mgr.smithi107.adejht (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:04.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 -- 172.21.15.115:0/2904437520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 msgr2=0x7f047c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 --2- 172.21.15.115:0/2904437520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c0f7d00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0464009fb0 tx=0x7f0464009b30).stop 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 -- 172.21.15.115:0/2904437520 shutdown_connections 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 --2- 172.21.15.115:0/2904437520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c0f7d00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 -- 172.21.15.115:0/2904437520 >> 172.21.15.115:0/2904437520 conn(0x7f047c0f0120 msgr2=0x7f047c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 -- 172.21.15.115:0/2904437520 shutdown_connections 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 -- 172.21.15.115:0/2904437520 wait complete. 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.869+0000 7f04815a2700 1 Processor -- start 2024-01-21T07:32:04.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f04815a2700 1 -- start start 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f04815a2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f04815a2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c108bc0 con 0x7f047c0f78e0 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f047bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f047bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54372/0 (socket says 172.21.15.115:54372) 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f047bfff700 1 -- 172.21.15.115:0/3829574778 learned_addr learned my addr 172.21.15.115:0/3829574778 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f047bfff700 1 -- 172.21.15.115:0/3829574778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464005040 con 0x7f047c0f78e0 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.870+0000 7f047bfff700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0464013510 tx=0x7f046400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:04.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0464009320 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c108dc0 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0464013f00 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0464014dc0 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c109250 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.871+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f046401d3d0 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.872+0000 7f0478ff9700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 0x7f0468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.872+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0464041100 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.872+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c045ad0 con 0x7f047c0f78e0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.872+0000 7f047b7fe700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 0x7f0468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:04.875 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.872+0000 7f047b7fe700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 0x7f0468032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f046c0096a0 tx=0x7f046c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:04.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:04.877+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0464012920 con 0x7f047c0f78e0 2024-01-21T07:32:05.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.083+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f047c109f70 con 0x7f047c0f78e0 2024-01-21T07:32:05.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.083+0000 7f0478ff9700 1 -- 172.21.15.115:0/3829574778 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0464012330 con 0x7f047c0f78e0 2024-01-21T07:32:05.086 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:05.086 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:05.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 msgr2=0x7f0468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:05.087 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 0x7f0468032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f046c0096a0 tx=0x7f046c006b40).stop 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 msgr2=0x7f047c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0464013510 tx=0x7f046400b040).stop 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 shutdown_connections 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f046802ff60 0x7f0468032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 --2- 172.21.15.115:0/3829574778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f047c0f78e0 0x7f047c1086d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 >> 172.21.15.115:0/3829574778 conn(0x7f047c0f0120 msgr2=0x7f047c0f0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 shutdown_connections 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:05.085+0000 7f04815a2700 1 -- 172.21.15.115:0/3829574778 wait complete. 2024-01-21T07:32:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:05.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:05.084932+0000 mon.smithi107 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.115:0/3829574778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:06.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:06.474 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:06.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:05.837030+0000 mgr.smithi107.adejht (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 -- 172.21.15.115:0/966981117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 msgr2=0x7efee40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 --2- 172.21.15.115:0/966981117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee40f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7efed4009fb0 tx=0x7efed4009b30).stop 2024-01-21T07:32:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 -- 172.21.15.115:0/966981117 shutdown_connections 2024-01-21T07:32:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 --2- 172.21.15.115:0/966981117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee40f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 -- 172.21.15.115:0/966981117 >> 172.21.15.115:0/966981117 conn(0x7efee40f1c70 msgr2=0x7efee40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 -- 172.21.15.115:0/966981117 shutdown_connections 2024-01-21T07:32:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.626+0000 7efeec2b4700 1 -- 172.21.15.115:0/966981117 wait complete. 2024-01-21T07:32:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeec2b4700 1 Processor -- start 2024-01-21T07:32:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeec2b4700 1 -- start start 2024-01-21T07:32:07.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeec2b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeec2b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee4106c20 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeea050700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeea050700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54398/0 (socket says 172.21.15.115:54398) 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeea050700 1 -- 172.21.15.115:0/3579533602 learned_addr learned my addr 172.21.15.115:0/3579533602 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.627+0000 7efeea050700 1 -- 172.21.15.115:0/3579533602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4005040 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efeea050700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7efed4013510 tx=0x7efed400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed4009320 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee4106e20 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efed4013f00 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed4014d90 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee41072b0 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.628+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efed4014420 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.629+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee4045ad0 con 0x7efee40f6510 2024-01-21T07:32:07.634 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.629+0000 7efedaffd700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 0x7efed0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.629+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efed4041000 con 0x7efee40f6510 2024-01-21T07:32:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.629+0000 7efee984f700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 0x7efed0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.629+0000 7efee984f700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 0x7efed0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efee00096a0 tx=0x7efee0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:07.635 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.633+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efed40095f0 con 0x7efee40f6510 2024-01-21T07:32:07.840 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.838+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efee40fbf60 con 0x7efee40f6510 2024-01-21T07:32:07.841 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.839+0000 7efedaffd700 1 -- 172.21.15.115:0/3579533602 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efed401db70 con 0x7efee40f6510 2024-01-21T07:32:07.841 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:07.842 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 msgr2=0x7efed0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 0x7efed0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efee00096a0 tx=0x7efee0006b40).stop 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 msgr2=0x7efee4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7efed4013510 tx=0x7efed400b040).stop 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 shutdown_connections 2024-01-21T07:32:07.843 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efed002ff60 0x7efed0032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 --2- 172.21.15.115:0/3579533602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efee40f6510 0x7efee4106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 >> 172.21.15.115:0/3579533602 conn(0x7efee40f1c70 msgr2=0x7efee40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 shutdown_connections 2024-01-21T07:32:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:07.840+0000 7efeec2b4700 1 -- 172.21.15.115:0/3579533602 wait complete. 2024-01-21T07:32:07.844 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:07.837392+0000 mgr.smithi107.adejht (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:07.840326+0000 mon.smithi107 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.115:0/3579533602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:09.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:09.217 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:09.837666+0000 mgr.smithi107.adejht (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 -- 172.21.15.115:0/303272300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 msgr2=0x7f6d2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 --2- 172.21.15.115:0/303272300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c0f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f6d28009fb0 tx=0x7f6d28009b30).stop 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 -- 172.21.15.115:0/303272300 shutdown_connections 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 --2- 172.21.15.115:0/303272300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c0f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 -- 172.21.15.115:0/303272300 >> 172.21.15.115:0/303272300 conn(0x7f6d2c0f09a0 msgr2=0x7f6d2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 -- 172.21.15.115:0/303272300 shutdown_connections 2024-01-21T07:32:10.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 -- 172.21.15.115:0/303272300 wait complete. 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.358+0000 7f6d32317700 1 Processor -- start 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d32317700 1 -- start start 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d32317700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d32317700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c060240 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d31315700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d31315700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47638/0 (socket says 172.21.15.115:47638) 2024-01-21T07:32:10.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d31315700 1 -- 172.21.15.115:0/1380123826 learned_addr learned my addr 172.21.15.115:0/1380123826 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:10.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d31315700 1 -- 172.21.15.115:0/1380123826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28005040 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.359+0000 7f6d31315700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6d28013510 tx=0x7f6d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.360+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d28009320 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.360+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c060440 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.360+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d28013f00 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.360+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d28014dc0 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.360+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c109860 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d2801d3d0 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d21ffb700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 0x7f6d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d280410e0 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c045ad0 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d30b14700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 0x7f6d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:10.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.361+0000 7f6d30b14700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 0x7f6d18032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6d1c0096a0 tx=0x7f6d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:10.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.366+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d28012920 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.575 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.572+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d2c109f60 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.575 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.572+0000 7f6d21ffb700 1 -- 172.21.15.115:0/1380123826 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6d2802b020 con 0x7f6d2c0f51a0 2024-01-21T07:32:10.575 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:10.575 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 msgr2=0x7f6d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 0x7f6d18032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6d1c0096a0 tx=0x7f6d1c006b40).stop 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 msgr2=0x7f6d2c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6d28013510 tx=0x7f6d2800b040).stop 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 shutdown_connections 2024-01-21T07:32:10.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6d1802ff60 0x7f6d18032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 --2- 172.21.15.115:0/1380123826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d2c0f51a0 0x7f6d2c1091b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 >> 172.21.15.115:0/1380123826 conn(0x7f6d2c0f09a0 msgr2=0x7f6d2c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 shutdown_connections 2024-01-21T07:32:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:10.574+0000 7f6d32317700 1 -- 172.21.15.115:0/1380123826 wait complete. 2024-01-21T07:32:10.577 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:09.947326+0000 mon.smithi107 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:32:11.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:09.948599+0000 mon.smithi107 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:32:11.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:10.573580+0000 mon.smithi107 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.115:0/1380123826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:11.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:11.965 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:13.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:11.838007+0000 mgr.smithi107.adejht (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:13.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 -- 172.21.15.115:0/257340540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 msgr2=0x7f7de40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 --2- 172.21.15.115:0/257340540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de40f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7dd4009fb0 tx=0x7f7dd4009b30).stop 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 -- 172.21.15.115:0/257340540 shutdown_connections 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 --2- 172.21.15.115:0/257340540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de40f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 -- 172.21.15.115:0/257340540 >> 172.21.15.115:0/257340540 conn(0x7f7de40f1c70 msgr2=0x7f7de40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 -- 172.21.15.115:0/257340540 shutdown_connections 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.074+0000 7f7debade700 1 -- 172.21.15.115:0/257340540 wait complete. 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7debade700 1 Processor -- start 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7debade700 1 -- start start 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7debade700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7debade700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de4069e20 con 0x7f7de40f6510 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7de987a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:13.080 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7de987a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47662/0 (socket says 172.21.15.115:47662) 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7de987a700 1 -- 172.21.15.115:0/1804092295 learned_addr learned my addr 172.21.15.115:0/1804092295 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7de987a700 1 -- 172.21.15.115:0/1804092295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd4005040 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.075+0000 7f7de987a700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7dd4013510 tx=0x7f7dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.076+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dd4009320 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.076+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de4069fc0 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.076+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dd4013f00 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.076+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dd4014d90 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.076+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de406a450 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7dd40095f0 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de4059100 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7dda7fc700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 0x7f7dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7dd4041000 con 0x7f7de40f6510 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7de9079700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 0x7f7dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:13.081 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.077+0000 7f7de9079700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 0x7f7dd0032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:13.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.082+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7dd4014420 con 0x7f7de40f6510 2024-01-21T07:32:13.288 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.286+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7de40fbf60 con 0x7f7de40f6510 2024-01-21T07:32:13.288 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.286+0000 7f7dda7fc700 1 -- 172.21.15.115:0/1804092295 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7dd401d7f0 con 0x7f7de40f6510 2024-01-21T07:32:13.289 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:13.289 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:13.290 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.287+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 msgr2=0x7f7dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:13.290 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 0x7f7dd0032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).stop 2024-01-21T07:32:13.290 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 msgr2=0x7f7de406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:13.290 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7dd4013510 tx=0x7f7dd400b040).stop 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 shutdown_connections 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7dd002ff60 0x7f7dd0032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 --2- 172.21.15.115:0/1804092295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de40f6510 0x7f7de406b9a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 >> 172.21.15.115:0/1804092295 conn(0x7f7de40f1c70 msgr2=0x7f7de40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 shutdown_connections 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:13.288+0000 7f7debade700 1 -- 172.21.15.115:0/1804092295 wait complete. 2024-01-21T07:32:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:14.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:13.287631+0000 mon.smithi107 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.115:0/1804092295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:14.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:14.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:15.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:13.838318+0000 mgr.smithi107.adejht (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:15.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 -- 172.21.15.115:0/173903770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 msgr2=0x7f22800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:15.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 --2- 172.21.15.115:0/173903770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f22800f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f227c009fb0 tx=0x7f227c009b30).stop 2024-01-21T07:32:15.814 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 -- 172.21.15.115:0/173903770 shutdown_connections 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 --2- 172.21.15.115:0/173903770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f22800f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 -- 172.21.15.115:0/173903770 >> 172.21.15.115:0/173903770 conn(0x7f22800f0b70 msgr2=0x7f22800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 -- 172.21.15.115:0/173903770 shutdown_connections 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 -- 172.21.15.115:0/173903770 wait complete. 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.811+0000 7f22877e4700 1 Processor -- start 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f22877e4700 1 -- start start 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f22877e4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f22877e4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2280105b20 con 0x7f22800f5410 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f2285580700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f2285580700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47682/0 (socket says 172.21.15.115:47682) 2024-01-21T07:32:15.815 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.812+0000 7f2285580700 1 -- 172.21.15.115:0/3448274060 learned_addr learned my addr 172.21.15.115:0/3448274060 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f2285580700 1 -- 172.21.15.115:0/3448274060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c005040 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f2285580700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f227c013510 tx=0x7f227c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c009320 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2280105d20 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f227c013f00 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227c014d90 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.813+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22801061b0 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.814+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f227c0095f0 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.814+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2280045ad0 con 0x7f22800f5410 2024-01-21T07:32:15.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.814+0000 7f2275ffb700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 0x7f226c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:15.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.814+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f227c0410c0 con 0x7f22800f5410 2024-01-21T07:32:15.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.814+0000 7f2284d7f700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 0x7f226c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:15.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.815+0000 7f2284d7f700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 0x7f226c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f22700096a0 tx=0x7f2270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:15.821 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:15.819+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f227c01d7f0 con 0x7f22800f5410 2024-01-21T07:32:16.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.024+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22800fae60 con 0x7f22800f5410 2024-01-21T07:32:16.027 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.026+0000 7f2275ffb700 1 -- 172.21.15.115:0/3448274060 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f227c010230 con 0x7f22800f5410 2024-01-21T07:32:16.028 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:16.028 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:16.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 msgr2=0x7f226c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:16.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 0x7f226c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f22700096a0 tx=0x7f2270006b40).stop 2024-01-21T07:32:16.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 msgr2=0x7f2280105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f227c013510 tx=0x7f227c00b040).stop 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 shutdown_connections 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f226c02ff60 0x7f226c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 --2- 172.21.15.115:0/3448274060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22800f5410 0x7f2280105630 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 >> 172.21.15.115:0/3448274060 conn(0x7f22800f0b70 msgr2=0x7f22800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 shutdown_connections 2024-01-21T07:32:16.030 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:16.027+0000 7f22877e4700 1 -- 172.21.15.115:0/3448274060 wait complete. 2024-01-21T07:32:16.031 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:15.838549+0000 mgr.smithi107.adejht (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:17.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:16.026354+0000 mon.smithi107 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.115:0/3448274060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:17.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:17.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:17.838822+0000 mgr.smithi107.adejht (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 -- 172.21.15.115:0/3531362568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 msgr2=0x7f1e500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 --2- 172.21.15.115:0/3531362568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e500f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f1e40009fb0 tx=0x7f1e40009b30).stop 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 -- 172.21.15.115:0/3531362568 shutdown_connections 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 --2- 172.21.15.115:0/3531362568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e500f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 -- 172.21.15.115:0/3531362568 >> 172.21.15.115:0/3531362568 conn(0x7f1e500f09a0 msgr2=0x7f1e500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 -- 172.21.15.115:0/3531362568 shutdown_connections 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 -- 172.21.15.115:0/3531362568 wait complete. 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.567+0000 7f1e553e7700 1 Processor -- start 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e553e7700 1 -- start start 2024-01-21T07:32:18.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e553e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:18.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e553e7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e501069b0 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e4ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47696/0 (socket says 172.21.15.115:47696) 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e4ffff700 1 -- 172.21.15.115:0/2010640327 learned_addr learned my addr 172.21.15.115:0/2010640327 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.568+0000 7f1e4ffff700 1 -- 172.21.15.115:0/2010640327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40005040 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e4ffff700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1e40013510 tx=0x7f1e4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e40009320 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e50106bb0 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e40013f00 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e40014dc0 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.569+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e50107040 con 0x7f1e500f51a0 2024-01-21T07:32:18.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e40014420 con 0x7f1e500f51a0 2024-01-21T07:32:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e50045ad0 con 0x7f1e500f51a0 2024-01-21T07:32:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e4cff9700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e40041030 con 0x7f1e500f51a0 2024-01-21T07:32:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e4f7fe700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:18.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.570+0000 7f1e4f7fe700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 0x7f1e3c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1e380096a0 tx=0x7f1e38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.575+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e400095f0 con 0x7f1e500f51a0 2024-01-21T07:32:18.783 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.780+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e50107ef0 con 0x7f1e500f51a0 2024-01-21T07:32:18.783 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.781+0000 7f1e4cff9700 1 -- 172.21.15.115:0/2010640327 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e4001cb10 con 0x7f1e500f51a0 2024-01-21T07:32:18.783 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:18.783 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 msgr2=0x7f1e3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 0x7f1e3c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1e380096a0 tx=0x7f1e38006b40).stop 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 msgr2=0x7f1e501064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1e40013510 tx=0x7f1e4000b040).stop 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 shutdown_connections 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:18.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 --2- 172.21.15.115:0/2010640327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e500f51a0 0x7f1e501064c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:18.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 >> 172.21.15.115:0/2010640327 conn(0x7f1e500f09a0 msgr2=0x7f1e500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:18.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 shutdown_connections 2024-01-21T07:32:18.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:18.783+0000 7f1e553e7700 1 -- 172.21.15.115:0/2010640327 wait complete. 2024-01-21T07:32:18.786 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:19.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:18.782561+0000 mon.smithi107 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.115:0/2010640327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:20.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:20.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:20.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:19.839167+0000 mgr.smithi107.adejht (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.292+0000 7f853a2fb700 1 -- 172.21.15.115:0/2610369137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 msgr2=0x7f85340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.292+0000 7f853a2fb700 1 --2- 172.21.15.115:0/2610369137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8530009fb0 tx=0x7f8530009b30).stop 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.292+0000 7f853a2fb700 1 -- 172.21.15.115:0/2610369137 shutdown_connections 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.292+0000 7f853a2fb700 1 --2- 172.21.15.115:0/2610369137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.292+0000 7f853a2fb700 1 -- 172.21.15.115:0/2610369137 >> 172.21.15.115:0/2610369137 conn(0x7f85340f09a0 msgr2=0x7f85340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 -- 172.21.15.115:0/2610369137 shutdown_connections 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 -- 172.21.15.115:0/2610369137 wait complete. 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 Processor -- start 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 -- start start 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f853a2fb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85340fb5a0 con 0x7f85340f51a0 2024-01-21T07:32:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f85392f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f85392f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:57050/0 (socket says 172.21.15.115:57050) 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.293+0000 7f85392f9700 1 -- 172.21.15.115:0/3267651014 learned_addr learned my addr 172.21.15.115:0/3267651014 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f85392f9700 1 -- 172.21.15.115:0/3267651014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530005040 con 0x7f85340f51a0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f85392f9700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8530013510 tx=0x7f853000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8530009320 con 0x7f85340f51a0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85340fb740 con 0x7f85340f51a0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8530013f00 con 0x7f85340f51a0 2024-01-21T07:32:21.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8530014dc0 con 0x7f85340f51a0 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.294+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85340fbbd0 con 0x7f85340f51a0 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.295+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8530014420 con 0x7f85340f51a0 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.295+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8534059100 con 0x7f85340f51a0 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.295+0000 7f8529ffb700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.295+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85300410f0 con 0x7f85340f51a0 2024-01-21T07:32:21.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.295+0000 7f8538af8700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:21.300 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.296+0000 7f8538af8700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 0x7f8520032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f85240096a0 tx=0x7f8524006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:21.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.300+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85300095f0 con 0x7f85340f51a0 2024-01-21T07:32:21.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.509+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85340fabe0 con 0x7f85340f51a0 2024-01-21T07:32:21.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.510+0000 7f8529ffb700 1 -- 172.21.15.115:0/3267651014 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f853001cb10 con 0x7f85340f51a0 2024-01-21T07:32:21.512 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:21.512 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 msgr2=0x7f8520032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 0x7f8520032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f85240096a0 tx=0x7f8524006b40).stop 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 msgr2=0x7f85340fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8530013510 tx=0x7f853000b040).stop 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 shutdown_connections 2024-01-21T07:32:21.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f852002ff60 0x7f8520032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:21.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 --2- 172.21.15.115:0/3267651014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85340f51a0 0x7f85340fd120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:21.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 >> 172.21.15.115:0/3267651014 conn(0x7f85340f09a0 msgr2=0x7f853405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:21.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 shutdown_connections 2024-01-21T07:32:21.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:21.512+0000 7f853a2fb700 1 -- 172.21.15.115:0/3267651014 wait complete. 2024-01-21T07:32:21.515 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:21.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:21.511253+0000 mon.smithi107 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.115:0/3267651014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:22.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:22.887 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:23.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:21.839549+0000 mgr.smithi107.adejht (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:23.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 -- 172.21.15.115:0/3994080758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 msgr2=0x7f20e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/3994080758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e80f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f20dc009fb0 tx=0x7f20dc009b30).stop 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 -- 172.21.15.115:0/3994080758 shutdown_connections 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/3994080758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e80f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 -- 172.21.15.115:0/3994080758 >> 172.21.15.115:0/3994080758 conn(0x7f20e80f0b70 msgr2=0x7f20e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 -- 172.21.15.115:0/3994080758 shutdown_connections 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.988+0000 7f20eeb4b700 1 -- 172.21.15.115:0/3994080758 wait complete. 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20eeb4b700 1 Processor -- start 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20eeb4b700 1 -- start start 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20eeb4b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20eeb4b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8060280 con 0x7f20e80f5410 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20ec8e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20ec8e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:57068/0 (socket says 172.21.15.115:57068) 2024-01-21T07:32:23.992 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20ec8e7700 1 -- 172.21.15.115:0/1524528697 learned_addr learned my addr 172.21.15.115:0/1524528697 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:23.993 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.989+0000 7f20ec8e7700 1 -- 172.21.15.115:0/1524528697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc005040 con 0x7f20e80f5410 2024-01-21T07:32:23.993 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20ec8e7700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f20dc013510 tx=0x7f20dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc009320 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e8060480 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e8109b20 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20dc013f00 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.990+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20dc014d60 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.991+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20dc0263a0 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.991+0000 7f20e57fa700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 0x7f20d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.991+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c8004fa0 con 0x7f20e80f5410 2024-01-21T07:32:23.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.991+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20dc041010 con 0x7f20e80f5410 2024-01-21T07:32:23.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.991+0000 7f20e7fff700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 0x7f20d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:23.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.992+0000 7f20e7fff700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 0x7f20d8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f20d40096a0 tx=0x7f20d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:23.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:23.996+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20dc01d380 con 0x7f20e80f5410 2024-01-21T07:32:24.207 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.204+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20c8005670 con 0x7f20e80f5410 2024-01-21T07:32:24.207 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.205+0000 7f20e57fa700 1 -- 172.21.15.115:0/1524528697 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f20dc02b020 con 0x7f20e80f5410 2024-01-21T07:32:24.208 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:24.208 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.206+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 msgr2=0x7f20d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.206+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 0x7f20d8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f20d40096a0 tx=0x7f20d4006b40).stop 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.206+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 msgr2=0x7f20e81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.206+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f20dc013510 tx=0x7f20dc00b040).stop 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 shutdown_connections 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f20d802ff60 0x7f20d8032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:24.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 --2- 172.21.15.115:0/1524528697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20e80f5410 0x7f20e81093f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:24.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 >> 172.21.15.115:0/1524528697 conn(0x7f20e80f0b70 msgr2=0x7f20e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:24.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 shutdown_connections 2024-01-21T07:32:24.210 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:24.207+0000 7f20eeb4b700 1 -- 172.21.15.115:0/1524528697 wait complete. 2024-01-21T07:32:24.210 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:23.839910+0000 mgr.smithi107.adejht (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:24.206428+0000 mon.smithi107 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.115:0/1524528697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:25.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:25.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:25.840209+0000 mgr.smithi107.adejht (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 -- 172.21.15.115:0/3827489896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 msgr2=0x7fbf7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3827489896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c0f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fbf6c009fb0 tx=0x7fbf6c009b30).stop 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 -- 172.21.15.115:0/3827489896 shutdown_connections 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3827489896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c0f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 -- 172.21.15.115:0/3827489896 >> 172.21.15.115:0/3827489896 conn(0x7fbf7c0f0b70 msgr2=0x7fbf7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 -- 172.21.15.115:0/3827489896 shutdown_connections 2024-01-21T07:32:26.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.717+0000 7fbf846c5700 1 -- 172.21.15.115:0/3827489896 wait complete. 2024-01-21T07:32:26.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.718+0000 7fbf846c5700 1 Processor -- start 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.718+0000 7fbf846c5700 1 -- start start 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.718+0000 7fbf846c5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.718+0000 7fbf846c5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c105af0 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.718+0000 7fbf82461700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf82461700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:57082/0 (socket says 172.21.15.115:57082) 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf82461700 1 -- 172.21.15.115:0/3978799684 learned_addr learned my addr 172.21.15.115:0/3978799684 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf82461700 1 -- 172.21.15.115:0/3978799684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c005040 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf82461700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fbf6c013510 tx=0x7fbf6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6c009320 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf6c013f00 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7c105cf0 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6c014d90 con 0x7fbf7c0f5410 2024-01-21T07:32:26.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.719+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7c106180 con 0x7fbf7c0f5410 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.720+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf6c0095f0 con 0x7fbf7c0f5410 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.720+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7c045ad0 con 0x7fbf7c0f5410 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.720+0000 7fbf72ffd700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.720+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf6c041000 con 0x7fbf7c0f5410 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.720+0000 7fbf81c60700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:26.724 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.721+0000 7fbf81c60700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 0x7fbf68032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf780096a0 tx=0x7fbf78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:26.727 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.725+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf6c014420 con 0x7fbf7c0f5410 2024-01-21T07:32:26.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.930+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf7c0fae60 con 0x7fbf7c0f5410 2024-01-21T07:32:26.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.931+0000 7fbf72ffd700 1 -- 172.21.15.115:0/3978799684 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf6c01d7f0 con 0x7fbf7c0f5410 2024-01-21T07:32:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:26.933 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:26.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 msgr2=0x7fbf68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 0x7fbf68032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf780096a0 tx=0x7fbf78006b40).stop 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 msgr2=0x7fbf7c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fbf6c013510 tx=0x7fbf6c00b040).stop 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 shutdown_connections 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 --2- 172.21.15.115:0/3978799684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf7c0f5410 0x7fbf7c105600 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 >> 172.21.15.115:0/3978799684 conn(0x7fbf7c0f0b70 msgr2=0x7fbf7c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 shutdown_connections 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:26.932+0000 7fbf846c5700 1 -- 172.21.15.115:0/3978799684 wait complete. 2024-01-21T07:32:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:28.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:26.932241+0000 mon.smithi107 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.115:0/3978799684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:28.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:28.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:29.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:27.840529+0000 mgr.smithi107.adejht (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 -- 172.21.15.115:0/4176682081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 msgr2=0x7f33700f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:29.468 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 --2- 172.21.15.115:0/4176682081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33700f9170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f3360009fb0 tx=0x7f3360009b30).stop 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 -- 172.21.15.115:0/4176682081 shutdown_connections 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 --2- 172.21.15.115:0/4176682081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33700f9170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 -- 172.21.15.115:0/4176682081 >> 172.21.15.115:0/4176682081 conn(0x7f33700f0b70 msgr2=0x7f33700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.465+0000 7f33767cd700 1 -- 172.21.15.115:0/4176682081 shutdown_connections 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.466+0000 7f33767cd700 1 -- 172.21.15.115:0/4176682081 wait complete. 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.466+0000 7f33767cd700 1 Processor -- start 2024-01-21T07:32:29.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.466+0000 7f33767cd700 1 -- start start 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.466+0000 7f33767cd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.466+0000 7f33767cd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370106b90 con 0x7f3370060200 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:38452/0 (socket says 172.21.15.115:38452) 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336ffff700 1 -- 172.21.15.115:0/399747713 learned_addr learned my addr 172.21.15.115:0/399747713 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336ffff700 1 -- 172.21.15.115:0/399747713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360005040 con 0x7f3370060200 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336ffff700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3360013510 tx=0x7f336000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360009320 con 0x7f3370060200 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3370106d90 con 0x7f3370060200 2024-01-21T07:32:29.473 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3360013f00 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360014dc0 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.467+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370107220 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.468+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f336001d3d0 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.468+0000 7f336cff9700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 0x7f3358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.468+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3370045ad0 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.468+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3360041130 con 0x7f3370060200 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.468+0000 7f336f7fe700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 0x7f3358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:29.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.469+0000 7f336f7fe700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 0x7f3358032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f33640096a0 tx=0x7f3364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:29.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.474+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f336001cb10 con 0x7f3370060200 2024-01-21T07:32:29.682 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.680+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33701080d0 con 0x7f3370060200 2024-01-21T07:32:29.682 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.680+0000 7f336cff9700 1 -- 172.21.15.115:0/399747713 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f336001cb10 con 0x7f3370060200 2024-01-21T07:32:29.683 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:29.683 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 msgr2=0x7f3358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 0x7f3358032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f33640096a0 tx=0x7f3364006b40).stop 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 msgr2=0x7f33701066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3360013510 tx=0x7f336000b040).stop 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 shutdown_connections 2024-01-21T07:32:29.684 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f335802ff60 0x7f3358032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:29.685 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 --2- 172.21.15.115:0/399747713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3370060200 0x7f33701066a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:29.685 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 >> 172.21.15.115:0/399747713 conn(0x7f33700f0b70 msgr2=0x7f33700f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:29.685 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 shutdown_connections 2024-01-21T07:32:29.685 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:29.683+0000 7f33767cd700 1 -- 172.21.15.115:0/399747713 wait complete. 2024-01-21T07:32:29.685 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:30.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:29.681658+0000 mon.smithi107 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.115:0/399747713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:31.038 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:31.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:31.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:29.840851+0000 mgr.smithi107.adejht (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- 172.21.15.115:0/3400056969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 msgr2=0x7f86180f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 --2- 172.21.15.115:0/3400056969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86180f7d00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8608009fb0 tx=0x7f8608009b30).stop 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- 172.21.15.115:0/3400056969 shutdown_connections 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 --2- 172.21.15.115:0/3400056969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86180f7d00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- 172.21.15.115:0/3400056969 >> 172.21.15.115:0/3400056969 conn(0x7f86180f0120 msgr2=0x7f86180f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- 172.21.15.115:0/3400056969 shutdown_connections 2024-01-21T07:32:32.157 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- 172.21.15.115:0/3400056969 wait complete. 2024-01-21T07:32:32.158 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 Processor -- start 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.155+0000 7f861759e700 1 -- start start 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861759e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181069d0 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:38462/0 (socket says 172.21.15.115:38462) 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861659c700 1 -- 172.21.15.115:0/496185896 learned_addr learned my addr 172.21.15.115:0/496185896 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861659c700 1 -- 172.21.15.115:0/496185896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608005040 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861659c700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f8608013510 tx=0x7f860800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8608009320 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8608013f00 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8618106bd0 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8608014dc0 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.156+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8618107060 con 0x7f86180f78e0 2024-01-21T07:32:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.157+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8608014420 con 0x7f86180f78e0 2024-01-21T07:32:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.157+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8618045ad0 con 0x7f86180f78e0 2024-01-21T07:32:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.157+0000 7f860effd700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.157+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8608041110 con 0x7f86180f78e0 2024-01-21T07:32:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.158+0000 7f8615d9b700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.158+0000 7f8615d9b700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 0x7f8604032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f86000096a0 tx=0x7f8600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:32.163 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.162+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86080095f0 con 0x7f86180f78e0 2024-01-21T07:32:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.372+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8618107f10 con 0x7f86180f78e0 2024-01-21T07:32:32.374 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.372+0000 7f860effd700 1 -- 172.21.15.115:0/496185896 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f860801cb10 con 0x7f86180f78e0 2024-01-21T07:32:32.374 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:32.374 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:32.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 msgr2=0x7f8604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:32.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 0x7f8604032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f86000096a0 tx=0x7f8600006b40).stop 2024-01-21T07:32:32.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 msgr2=0x7f86181064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f8608013510 tx=0x7f860800b040).stop 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 shutdown_connections 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 --2- 172.21.15.115:0/496185896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86180f78e0 0x7f86181064e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 >> 172.21.15.115:0/496185896 conn(0x7f86180f0120 msgr2=0x7f86180f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 shutdown_connections 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:32.374+0000 7f861759e700 1 -- 172.21.15.115:0/496185896 wait complete. 2024-01-21T07:32:32.376 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:31.841225+0000 mgr.smithi107.adejht (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:32.372809+0000 mon.smithi107 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.115:0/496185896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:33.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:33.742 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:33.841584+0000 mgr.smithi107.adejht (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 -- 172.21.15.115:0/1816600798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 msgr2=0x7f674c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 --2- 172.21.15.115:0/1816600798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f673c009fb0 tx=0x7f673c009b30).stop 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 -- 172.21.15.115:0/1816600798 shutdown_connections 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 --2- 172.21.15.115:0/1816600798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 -- 172.21.15.115:0/1816600798 >> 172.21.15.115:0/1816600798 conn(0x7f674c0f0b70 msgr2=0x7f674c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 -- 172.21.15.115:0/1816600798 shutdown_connections 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.873+0000 7f6753893700 1 -- 172.21.15.115:0/1816600798 wait complete. 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f6753893700 1 Processor -- start 2024-01-21T07:32:34.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f6753893700 1 -- start start 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f6753893700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f6753893700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c106c10 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f675162f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f675162f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:38480/0 (socket says 172.21.15.115:38480) 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f675162f700 1 -- 172.21.15.115:0/1936668197 learned_addr learned my addr 172.21.15.115:0/1936668197 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f675162f700 1 -- 172.21.15.115:0/1936668197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c005040 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.874+0000 7f675162f700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f673c013510 tx=0x7f673c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f673c009320 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c106e10 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f673c013f00 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f673c014d90 con 0x7f674c0f5410 2024-01-21T07:32:34.877 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c1072a0 con 0x7f674c0f5410 2024-01-21T07:32:34.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.875+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f673c0095f0 con 0x7f674c0f5410 2024-01-21T07:32:34.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.876+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c045ad0 con 0x7f674c0f5410 2024-01-21T07:32:34.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.876+0000 7f6741ffb700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 0x7f6738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:34.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.876+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f673c041080 con 0x7f674c0f5410 2024-01-21T07:32:34.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.876+0000 7f6750e2e700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 0x7f6738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:34.879 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.876+0000 7f6750e2e700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 0x7f6738032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f67480096a0 tx=0x7f6748006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:34.881 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:34.880+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f673c014420 con 0x7f674c0f5410 2024-01-21T07:32:35.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.086+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f674c108150 con 0x7f674c0f5410 2024-01-21T07:32:35.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.087+0000 7f6741ffb700 1 -- 172.21.15.115:0/1936668197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f673c01d7f0 con 0x7f674c0f5410 2024-01-21T07:32:35.088 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:35.089 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:35.090 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 msgr2=0x7f6738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 0x7f6738032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f67480096a0 tx=0x7f6748006b40).stop 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 msgr2=0x7f674c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f673c013510 tx=0x7f673c00b040).stop 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 shutdown_connections 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f673802ff60 0x7f6738032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 --2- 172.21.15.115:0/1936668197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f674c0f5410 0x7f674c106720 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 >> 172.21.15.115:0/1936668197 conn(0x7f674c0f0b70 msgr2=0x7f674c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 shutdown_connections 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:35.089+0000 7f6753893700 1 -- 172.21.15.115:0/1936668197 wait complete. 2024-01-21T07:32:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:35.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:35.087641+0000 mon.smithi107 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.115:0/1936668197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:36.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:36.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:36.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:35.841944+0000 mgr.smithi107.adejht (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:37.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- 172.21.15.115:0/4054477586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 msgr2=0x7f34a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:37.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 --2- 172.21.15.115:0/4054477586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a40f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f3494009fb0 tx=0x7f3494009b30).stop 2024-01-21T07:32:37.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- 172.21.15.115:0/4054477586 shutdown_connections 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 --2- 172.21.15.115:0/4054477586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a40f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- 172.21.15.115:0/4054477586 >> 172.21.15.115:0/4054477586 conn(0x7f34a40f0b70 msgr2=0x7f34a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- 172.21.15.115:0/4054477586 shutdown_connections 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- 172.21.15.115:0/4054477586 wait complete. 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 Processor -- start 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- start start 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.587+0000 7f34a9c65700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4106c10 con 0x7f34a40f5410 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a37fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:38492/0 (socket says 172.21.15.115:38492) 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a37fe700 1 -- 172.21.15.115:0/3546080026 learned_addr learned my addr 172.21.15.115:0/3546080026 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a37fe700 1 -- 172.21.15.115:0/3546080026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494005040 con 0x7f34a40f5410 2024-01-21T07:32:37.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a37fe700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3494014000 tx=0x7f349400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34940095f0 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4106e10 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3494007e70 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3494014ea0 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.588+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a41072a0 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.589+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3494025070 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.589+0000 7f34a0ff9700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.589+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a4045ad0 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.589+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3494040930 con 0x7f34a40f5410 2024-01-21T07:32:37.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.589+0000 7f349bfff700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:37.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.590+0000 7f349bfff700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f348c0096a0 tx=0x7f348c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:37.595 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.594+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f349400f300 con 0x7f34a40f5410 2024-01-21T07:32:37.802 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.801+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34a4108150 con 0x7f34a40f5410 2024-01-21T07:32:37.803 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.801+0000 7f34a0ff9700 1 -- 172.21.15.115:0/3546080026 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f349400f300 con 0x7f34a40f5410 2024-01-21T07:32:37.803 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:37.803 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:37.804 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 msgr2=0x7f3490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:37.804 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f348c0096a0 tx=0x7f348c006b40).stop 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 msgr2=0x7f34a4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3494014000 tx=0x7f349400b040).stop 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 shutdown_connections 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 --2- 172.21.15.115:0/3546080026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34a40f5410 0x7f34a4106720 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 >> 172.21.15.115:0/3546080026 conn(0x7f34a40f0b70 msgr2=0x7f34a405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 shutdown_connections 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:37.803+0000 7f34a9c65700 1 -- 172.21.15.115:0/3546080026 wait complete. 2024-01-21T07:32:37.805 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:37.801948+0000 mon.smithi107 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.115:0/3546080026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:39.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:39.204 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:37.842300+0000 mgr.smithi107.adejht (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 -- 172.21.15.115:0/2625956758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 msgr2=0x7f7f740f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 --2- 172.21.15.115:0/2625956758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f740f8f20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7f6c009fb0 tx=0x7f7f6c009b30).stop 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 -- 172.21.15.115:0/2625956758 shutdown_connections 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 --2- 172.21.15.115:0/2625956758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f740f8f20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 -- 172.21.15.115:0/2625956758 >> 172.21.15.115:0/2625956758 conn(0x7f7f740f09a0 msgr2=0x7f7f740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 -- 172.21.15.115:0/2625956758 shutdown_connections 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 -- 172.21.15.115:0/2625956758 wait complete. 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.327+0000 7f7f78905700 1 Processor -- start 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f78905700 1 -- start start 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f78905700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:40.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f78905700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f741069f0 con 0x7f7f740601e0 2024-01-21T07:32:40.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:40.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60386/0 (socket says 172.21.15.115:60386) 2024-01-21T07:32:40.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f737fe700 1 -- 172.21.15.115:0/4100786760 learned_addr learned my addr 172.21.15.115:0/4100786760 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:40.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.328+0000 7f7f737fe700 1 -- 172.21.15.115:0/4100786760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c005040 con 0x7f7f740601e0 2024-01-21T07:32:40.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f737fe700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7f6c013510 tx=0x7f7f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6c009320 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f74106bf0 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f6c013f00 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6c014d90 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.329+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f74107080 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.330+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f6c0095f0 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.330+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f74045ad0 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.330+0000 7f7f6bfff700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 0x7f7f54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.330+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f6c041000 con 0x7f7f740601e0 2024-01-21T07:32:40.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.330+0000 7f7f72ffd700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 0x7f7f54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:40.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.331+0000 7f7f72ffd700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 0x7f7f54032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7f5c0089e0 tx=0x7f7f5c006ec0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:40.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.335+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f6c01d7f0 con 0x7f7f740601e0 2024-01-21T07:32:40.545 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.544+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f74107f30 con 0x7f7f740601e0 2024-01-21T07:32:40.545 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.544+0000 7f7f6bfff700 1 -- 172.21.15.115:0/4100786760 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f6c010230 con 0x7f7f740601e0 2024-01-21T07:32:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:40.546 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 msgr2=0x7f7f54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 0x7f7f54032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7f5c0089e0 tx=0x7f7f5c006ec0).stop 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 msgr2=0x7f7f74106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7f6c013510 tx=0x7f7f6c00b040).stop 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 shutdown_connections 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7f5402ff60 0x7f7f54032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 --2- 172.21.15.115:0/4100786760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f740601e0 0x7f7f74106500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 >> 172.21.15.115:0/4100786760 conn(0x7f7f740f09a0 msgr2=0x7f7f740f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 shutdown_connections 2024-01-21T07:32:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:40.546+0000 7f7f78905700 1 -- 172.21.15.115:0/4100786760 wait complete. 2024-01-21T07:32:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:39.842565+0000 mgr.smithi107.adejht (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:40.544820+0000 mon.smithi107 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.115:0/4100786760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:41.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:41.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:43.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:41.842824+0000 mgr.smithi107.adejht (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:43.115 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 -- 172.21.15.115:0/1920742440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 msgr2=0x7f509c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:43.115 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 --2- 172.21.15.115:0/1920742440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f90f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f5090009fb0 tx=0x7f5090009b30).stop 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 -- 172.21.15.115:0/1920742440 shutdown_connections 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 --2- 172.21.15.115:0/1920742440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f90f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 -- 172.21.15.115:0/1920742440 >> 172.21.15.115:0/1920742440 conn(0x7f509c0f0b70 msgr2=0x7f509c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 -- 172.21.15.115:0/1920742440 shutdown_connections 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 -- 172.21.15.115:0/1920742440 wait complete. 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.114+0000 7f50a2463700 1 Processor -- start 2024-01-21T07:32:43.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f50a2463700 1 -- start start 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f50a2463700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f50a2463700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0f4ff0 con 0x7f509c060280 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f509bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f509bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60406/0 (socket says 172.21.15.115:60406) 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.115+0000 7f509bfff700 1 -- 172.21.15.115:0/1457431846 learned_addr learned my addr 172.21.15.115:0/1457431846 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:43.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f509bfff700 1 -- 172.21.15.115:0/1457431846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090005040 con 0x7f509c060280 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f509bfff700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5090013510 tx=0x7f509000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5090009320 con 0x7f509c060280 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c0f5190 con 0x7f509c060280 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5090013f00 con 0x7f509c060280 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5090014dc0 con 0x7f509c060280 2024-01-21T07:32:43.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.116+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c0f5620 con 0x7f509c060280 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.117+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5090014420 con 0x7f509c060280 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.117+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c059100 con 0x7f509c060280 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.117+0000 7f5098ff9700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 0x7f5084032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.117+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50900410a0 con 0x7f509c060280 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.117+0000 7f509b7fe700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 0x7f5084032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:43.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.118+0000 7f509b7fe700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 0x7f5084032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f508c0096a0 tx=0x7f508c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.122+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50900095f0 con 0x7f509c060280 2024-01-21T07:32:43.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.330+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f509c045ad0 con 0x7f509c060280 2024-01-21T07:32:43.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.331+0000 7f5098ff9700 1 -- 172.21.15.115:0/1457431846 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f509001cb10 con 0x7f509c060280 2024-01-21T07:32:43.332 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:43.332 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 msgr2=0x7f5084032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 0x7f5084032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f508c0096a0 tx=0x7f508c006b40).stop 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 msgr2=0x7f509c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5090013510 tx=0x7f509000b040).stop 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 shutdown_connections 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f508402ff60 0x7f5084032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 --2- 172.21.15.115:0/1457431846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f509c060280 0x7f509c0f6b70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 >> 172.21.15.115:0/1457431846 conn(0x7f509c0f0b70 msgr2=0x7f509c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:43.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 shutdown_connections 2024-01-21T07:32:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:43.333+0000 7f50a2463700 1 -- 172.21.15.115:0/1457431846 wait complete. 2024-01-21T07:32:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:44.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:43.331362+0000 mon.smithi107 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.115:0/1457431846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:44.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:44.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:45.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:43.843163+0000 mgr.smithi107.adejht (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 -- 172.21.15.115:0/1867629049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 msgr2=0x7fe0200f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 --2- 172.21.15.115:0/1867629049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0200f9150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fe010009fb0 tx=0x7fe010009b30).stop 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 -- 172.21.15.115:0/1867629049 shutdown_connections 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 --2- 172.21.15.115:0/1867629049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0200f9150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 -- 172.21.15.115:0/1867629049 >> 172.21.15.115:0/1867629049 conn(0x7fe0200f0b70 msgr2=0x7fe0200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 -- 172.21.15.115:0/1867629049 shutdown_connections 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.836+0000 7fe027333700 1 -- 172.21.15.115:0/1867629049 wait complete. 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe027333700 1 Processor -- start 2024-01-21T07:32:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe027333700 1 -- start start 2024-01-21T07:32:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe027333700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:45.840 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe027333700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe020106bd0 con 0x7fe020060220 2024-01-21T07:32:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe0250cf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:45.841 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe0250cf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60436/0 (socket says 172.21.15.115:60436) 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe0250cf700 1 -- 172.21.15.115:0/2193034452 learned_addr learned my addr 172.21.15.115:0/2193034452 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe0250cf700 1 -- 172.21.15.115:0/2193034452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010005040 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.837+0000 7fe0250cf700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe010013510 tx=0x7fe01000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.838+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe010009320 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.838+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe010013f00 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.838+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe020106dd0 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.838+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe010014dc0 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.838+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe020107260 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.839+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe010014420 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.839+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020045ad0 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.839+0000 7fe015ffb700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.839+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0100410f0 con 0x7fe020060220 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.839+0000 7fe0248ce700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:45.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.840+0000 7fe0248ce700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 0x7fe00c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe01c0096a0 tx=0x7fe01c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:45.846 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:45.844+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0100095f0 con 0x7fe020060220 2024-01-21T07:32:46.050 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.048+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe020108110 con 0x7fe020060220 2024-01-21T07:32:46.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.049+0000 7fe015ffb700 1 -- 172.21.15.115:0/2193034452 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe01001cb10 con 0x7fe020060220 2024-01-21T07:32:46.051 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:46.051 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 msgr2=0x7fe00c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 0x7fe00c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe01c0096a0 tx=0x7fe01c006b40).stop 2024-01-21T07:32:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 msgr2=0x7fe0201066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe010013510 tx=0x7fe01000b040).stop 2024-01-21T07:32:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 shutdown_connections 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe00c02ff60 0x7fe00c032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 --2- 172.21.15.115:0/2193034452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe020060220 0x7fe0201066e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 >> 172.21.15.115:0/2193034452 conn(0x7fe0200f0b70 msgr2=0x7fe0200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 shutdown_connections 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:46.051+0000 7fe027333700 1 -- 172.21.15.115:0/2193034452 wait complete. 2024-01-21T07:32:46.054 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:45.843367+0000 mgr.smithi107.adejht (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:46.050306+0000 mon.smithi107 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.115:0/2193034452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:47.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:47.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:47.843618+0000 mgr.smithi107.adejht (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 -- 172.21.15.115:0/474885656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 msgr2=0x7fba280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 --2- 172.21.15.115:0/474885656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba280f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fba1c009fb0 tx=0x7fba1c009b30).stop 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 -- 172.21.15.115:0/474885656 shutdown_connections 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 --2- 172.21.15.115:0/474885656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba280f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 -- 172.21.15.115:0/474885656 >> 172.21.15.115:0/474885656 conn(0x7fba280f0b70 msgr2=0x7fba280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 -- 172.21.15.115:0/474885656 shutdown_connections 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.583+0000 7fba2dec4700 1 -- 172.21.15.115:0/474885656 wait complete. 2024-01-21T07:32:48.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba2dec4700 1 Processor -- start 2024-01-21T07:32:48.587 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba2dec4700 1 -- start start 2024-01-21T07:32:48.587 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba2dec4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:48.587 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba2dec4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28105af0 con 0x7fba280f5410 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba277fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba277fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60452/0 (socket says 172.21.15.115:60452) 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.584+0000 7fba277fe700 1 -- 172.21.15.115:0/1234898445 learned_addr learned my addr 172.21.15.115:0/1234898445 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba277fe700 1 -- 172.21.15.115:0/1234898445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c005040 con 0x7fba280f5410 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba277fe700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c009320 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28105cf0 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba1c013f00 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c014d90 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.585+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba28106180 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.586+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba1c0263a0 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.586+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba28045ad0 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.586+0000 7fba0ffff700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 0x7fba10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.586+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba1c0412b0 con 0x7fba280f5410 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.586+0000 7fba26ffd700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 0x7fba10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.587+0000 7fba26ffd700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 0x7fba10032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fba180096a0 tx=0x7fba18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:48.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.591+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba1c01cad0 con 0x7fba280f5410 2024-01-21T07:32:48.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.796+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba280fae60 con 0x7fba280f5410 2024-01-21T07:32:48.799 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.797+0000 7fba0ffff700 1 -- 172.21.15.115:0/1234898445 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba1c01cad0 con 0x7fba280f5410 2024-01-21T07:32:48.799 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:48.799 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:48.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 msgr2=0x7fba10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 0x7fba10032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fba180096a0 tx=0x7fba18006b40).stop 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 msgr2=0x7fba28105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).stop 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 shutdown_connections 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fba1002ff60 0x7fba10032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 --2- 172.21.15.115:0/1234898445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba280f5410 0x7fba28105600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 >> 172.21.15.115:0/1234898445 conn(0x7fba280f0b70 msgr2=0x7fba280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.798+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 shutdown_connections 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:48.799+0000 7fba2dec4700 1 -- 172.21.15.115:0/1234898445 wait complete. 2024-01-21T07:32:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:48.798319+0000 mon.smithi107 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.115:0/1234898445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:50.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:50.179 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:50.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:49.843888+0000 mgr.smithi107.adejht (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:51.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 -- 172.21.15.115:0/2030813446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 msgr2=0x7f291c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:51.347 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 --2- 172.21.15.115:0/2030813446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c0f66c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f290c009fb0 tx=0x7f290c009b30).stop 2024-01-21T07:32:51.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 -- 172.21.15.115:0/2030813446 shutdown_connections 2024-01-21T07:32:51.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 --2- 172.21.15.115:0/2030813446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c0f66c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:51.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 -- 172.21.15.115:0/2030813446 >> 172.21.15.115:0/2030813446 conn(0x7f291c0f1aa0 msgr2=0x7f291c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:51.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 -- 172.21.15.115:0/2030813446 shutdown_connections 2024-01-21T07:32:51.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.345+0000 7f292095f700 1 -- 172.21.15.115:0/2030813446 wait complete. 2024-01-21T07:32:51.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f292095f700 1 Processor -- start 2024-01-21T07:32:51.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f292095f700 1 -- start start 2024-01-21T07:32:51.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f292095f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:51.350 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f292095f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1069a0 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f291ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f291ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51164/0 (socket says 172.21.15.115:51164) 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.346+0000 7f291ad9d700 1 -- 172.21.15.115:0/3589787504 learned_addr learned my addr 172.21.15.115:0/3589787504 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f291ad9d700 1 -- 172.21.15.115:0/3589787504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c005040 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f291ad9d700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f290c013510 tx=0x7f290c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290c009320 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c106ba0 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f290c013f00 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290c014dc0 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.347+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c107030 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f290c014420 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c045ad0 con 0x7f291c0f62a0 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f29037fe700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:51.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f290c041020 con 0x7f291c0f62a0 2024-01-21T07:32:51.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f291a59c700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:51.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.348+0000 7f291a59c700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 0x7f2904032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f29100096a0 tx=0x7f2910006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:51.354 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.352+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f290c0095f0 con 0x7f291c0f62a0 2024-01-21T07:32:51.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.557+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f291c107e80 con 0x7f291c0f62a0 2024-01-21T07:32:51.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.558+0000 7f29037fe700 1 -- 172.21.15.115:0/3589787504 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f290c01cb10 con 0x7f291c0f62a0 2024-01-21T07:32:51.560 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:51.561 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 msgr2=0x7f2904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 0x7f2904032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f29100096a0 tx=0x7f2910006b40).stop 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 msgr2=0x7f291c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f290c013510 tx=0x7f290c00b040).stop 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 shutdown_connections 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 --2- 172.21.15.115:0/3589787504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f291c0f62a0 0x7f291c1064b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 >> 172.21.15.115:0/3589787504 conn(0x7f291c0f1aa0 msgr2=0x7f291c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:51.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 shutdown_connections 2024-01-21T07:32:51.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:51.559+0000 7f292095f700 1 -- 172.21.15.115:0/3589787504 wait complete. 2024-01-21T07:32:51.563 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:52.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:51.559432+0000 mon.smithi107 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.115:0/3589787504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:52.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:52.956 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:53.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:51.844142+0000 mgr.smithi107.adejht (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 -- 172.21.15.115:0/4195139635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 msgr2=0x7f0fa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 --2- 172.21.15.115:0/4195139635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa40f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f0f94009fb0 tx=0x7f0f94009b30).stop 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 -- 172.21.15.115:0/4195139635 shutdown_connections 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 --2- 172.21.15.115:0/4195139635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa40f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 -- 172.21.15.115:0/4195139635 >> 172.21.15.115:0/4195139635 conn(0x7f0fa40f0b70 msgr2=0x7f0fa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 -- 172.21.15.115:0/4195139635 shutdown_connections 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.115+0000 7f0fab630700 1 -- 172.21.15.115:0/4195139635 wait complete. 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fab630700 1 Processor -- start 2024-01-21T07:32:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fab630700 1 -- start start 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fab630700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fab630700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa4060240 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fa93cc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fa93cc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51196/0 (socket says 172.21.15.115:51196) 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fa93cc700 1 -- 172.21.15.115:0/575722847 learned_addr learned my addr 172.21.15.115:0/575722847 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.116+0000 7f0fa93cc700 1 -- 172.21.15.115:0/575722847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94005040 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0fa93cc700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f94013510 tx=0x7f0f9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f94009320 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa4060440 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f94013f00 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f94014d90 con 0x7f0fa40f5410 2024-01-21T07:32:54.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.117+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa4109ae0 con 0x7f0fa40f5410 2024-01-21T07:32:54.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f940263a0 con 0x7f0fa40f5410 2024-01-21T07:32:54.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0f99ffb700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 0x7f0f90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:54.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f940410c0 con 0x7f0fa40f5410 2024-01-21T07:32:54.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa4045ad0 con 0x7f0fa40f5410 2024-01-21T07:32:54.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0fa8bcb700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 0x7f0f90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:54.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.118+0000 7f0fa8bcb700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 0x7f0f90032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:54.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.123+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f94012920 con 0x7f0fa40f5410 2024-01-21T07:32:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:53.844454+0000 mgr.smithi107.adejht (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:54.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.328+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fa40faf50 con 0x7f0fa40f5410 2024-01-21T07:32:54.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.328+0000 7f0f99ffb700 1 -- 172.21.15.115:0/575722847 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f9402b020 con 0x7f0fa40f5410 2024-01-21T07:32:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:54.331 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 msgr2=0x7f0f90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 0x7f0f90032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).stop 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 msgr2=0x7f0fa4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f94013510 tx=0x7f0f9400b040).stop 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 shutdown_connections 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0f9002ff60 0x7f0f90032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 --2- 172.21.15.115:0/575722847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fa40f5410 0x7f0fa4109430 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:54.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 >> 172.21.15.115:0/575722847 conn(0x7f0fa40f0b70 msgr2=0x7f0fa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:54.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 shutdown_connections 2024-01-21T07:32:54.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:54.330+0000 7f0fab630700 1 -- 172.21.15.115:0/575722847 wait complete. 2024-01-21T07:32:54.333 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:54.329724+0000 mon.smithi107 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.115:0/575722847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:55.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:55.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:55.844666+0000 mgr.smithi107.adejht (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:56.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.864+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3064250378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 msgr2=0x7fdd340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.864+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3064250378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd340f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fdd28009fb0 tx=0x7fdd28009b30).stop 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.864+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3064250378 shutdown_connections 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.864+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3064250378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd340f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.864+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3064250378 >> 172.21.15.115:0/3064250378 conn(0x7fdd340f1c70 msgr2=0x7fdd340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3064250378 shutdown_connections 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3064250378 wait complete. 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 Processor -- start 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 -- start start 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.865+0000 7fdd3ad01700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd34106c20 con 0x7fdd340f6510 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd38a9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd38a9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51208/0 (socket says 172.21.15.115:51208) 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd38a9d700 1 -- 172.21.15.115:0/3727680176 learned_addr learned my addr 172.21.15.115:0/3727680176 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd38a9d700 1 -- 172.21.15.115:0/3727680176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28005040 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd38a9d700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdd28013510 tx=0x7fdd2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd28009320 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd34106e20 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd28013f00 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd28014d90 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.866+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd341072b0 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.867+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd280263a0 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.867+0000 7fdd317fa700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.867+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd28040ff0 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.867+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd34045ad0 con 0x7fdd340f6510 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.867+0000 7fdd33fff700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:56.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.868+0000 7fdd33fff700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 0x7fdd24032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdd200096a0 tx=0x7fdd20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:56.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:56.872+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd28012920 con 0x7fdd340f6510 2024-01-21T07:32:57.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.081+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd340fc050 con 0x7fdd340f6510 2024-01-21T07:32:57.084 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.082+0000 7fdd317fa700 1 -- 172.21.15.115:0/3727680176 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd2802b020 con 0x7fdd340f6510 2024-01-21T07:32:57.084 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:57.084 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:57.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 msgr2=0x7fdd24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 0x7fdd24032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdd200096a0 tx=0x7fdd20006b40).stop 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 msgr2=0x7fdd34106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdd28013510 tx=0x7fdd2800b040).stop 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 shutdown_connections 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 --2- 172.21.15.115:0/3727680176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd340f6510 0x7fdd34106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 >> 172.21.15.115:0/3727680176 conn(0x7fdd340f1c70 msgr2=0x7fdd340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 shutdown_connections 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:57.083+0000 7fdd3ad01700 1 -- 172.21.15.115:0/3727680176 wait complete. 2024-01-21T07:32:57.086 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:32:58.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:57.083202+0000 mon.smithi107 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.115:0/3727680176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:32:58.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:32:58.458 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:32:59.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:57.844906+0000 mgr.smithi107.adejht (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 -- 172.21.15.115:0/1046544964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 msgr2=0x7fa9c00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/1046544964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c00f4b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa9b4009fb0 tx=0x7fa9b4009b30).stop 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 -- 172.21.15.115:0/1046544964 shutdown_connections 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/1046544964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c00f4b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 -- 172.21.15.115:0/1046544964 >> 172.21.15.115:0/1046544964 conn(0x7fa9c00f0260 msgr2=0x7fa9c00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 -- 172.21.15.115:0/1046544964 shutdown_connections 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 -- 172.21.15.115:0/1046544964 wait complete. 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.568+0000 7fa9c62ad700 1 Processor -- start 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9c62ad700 1 -- start start 2024-01-21T07:32:59.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9c62ad700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:59.572 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9c62ad700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c0108db0 con 0x7fa9c00f4740 2024-01-21T07:32:59.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9bffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:59.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9bffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36528/0 (socket says 172.21.15.115:36528) 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.569+0000 7fa9bffff700 1 -- 172.21.15.115:0/217528652 learned_addr learned my addr 172.21.15.115:0/217528652 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9bffff700 1 -- 172.21.15.115:0/217528652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4005040 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9bffff700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa9b4013510 tx=0x7fa9b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b4009320 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c0108fb0 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c0109440 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9b4013f00 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.570+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b4014cc0 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.571+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9b4014420 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.571+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a0004fa0 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.571+0000 7fa9bcff9700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 0x7fa9a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.571+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9b4041110 con 0x7fa9c00f4740 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.571+0000 7fa9bf7fe700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 0x7fa9a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:32:59.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.572+0000 7fa9bf7fe700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 0x7fa9a8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa9b00096a0 tx=0x7fa9b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:32:59.578 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.576+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9b4012b60 con 0x7fa9c00f4740 2024-01-21T07:32:59.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.784+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9a0005670 con 0x7fa9c00f4740 2024-01-21T07:32:59.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.785+0000 7fa9bcff9700 1 -- 172.21.15.115:0/217528652 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9b4012980 con 0x7fa9c00f4740 2024-01-21T07:32:59.788 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:32:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:32:59.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 msgr2=0x7fa9a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:59.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 0x7fa9a8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa9b00096a0 tx=0x7fa9b0006b40).stop 2024-01-21T07:32:59.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 msgr2=0x7fa9c01088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:32:59.789 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa9b4013510 tx=0x7fa9b400b040).stop 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 shutdown_connections 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa9a802ffb0 0x7fa9a8032470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 --2- 172.21.15.115:0/217528652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9c00f4740 0x7fa9c01088c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 >> 172.21.15.115:0/217528652 conn(0x7fa9c00f0260 msgr2=0x7fa9c00f10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 shutdown_connections 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:32:59.787+0000 7fa9c62ad700 1 -- 172.21.15.115:0/217528652 wait complete. 2024-01-21T07:32:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:00.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:32:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:32:59.786628+0000 mon.smithi107 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.115:0/217528652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:01.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:32:59.845263+0000 mgr.smithi107.adejht (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:01.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:01.179 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:02.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- 172.21.15.115:0/3048503770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 msgr2=0x7f03380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 --2- 172.21.15.115:0/3048503770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f0328009fb0 tx=0x7f0328009b30).stop 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- 172.21.15.115:0/3048503770 shutdown_connections 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 --2- 172.21.15.115:0/3048503770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- 172.21.15.115:0/3048503770 >> 172.21.15.115:0/3048503770 conn(0x7f03380f09a0 msgr2=0x7f03380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- 172.21.15.115:0/3048503770 shutdown_connections 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- 172.21.15.115:0/3048503770 wait complete. 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 Processor -- start 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- start start 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:02.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.359+0000 7f033cca9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03380fb5a0 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f03377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f03377fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36546/0 (socket says 172.21.15.115:36546) 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f03377fe700 1 -- 172.21.15.115:0/164942868 learned_addr learned my addr 172.21.15.115:0/164942868 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f03377fe700 1 -- 172.21.15.115:0/164942868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328005040 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f03377fe700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0328009830 tx=0x7f032800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0328009320 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03380fb740 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03280136c0 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032801ee90 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.360+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03380fbbd0 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.361+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f032801e420 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.361+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0338059100 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.361+0000 7f031ffff700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 0x7f03200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.361+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0328040fe0 con 0x7f03380f51a0 2024-01-21T07:33:02.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.361+0000 7f0336ffd700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 0x7f03200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:02.366 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.362+0000 7f0336ffd700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 0x7f03200323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f032c0096a0 tx=0x7f032c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:02.368 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.366+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03280095f0 con 0x7f03380f51a0 2024-01-21T07:33:02.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.571+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03380fabe0 con 0x7f03380f51a0 2024-01-21T07:33:02.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.572+0000 7f031ffff700 1 -- 172.21.15.115:0/164942868 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f032802c2d0 con 0x7f03380f51a0 2024-01-21T07:33:02.574 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:02.574 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 msgr2=0x7f03200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 0x7f03200323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f032c0096a0 tx=0x7f032c006b40).stop 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 msgr2=0x7f03380fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0328009830 tx=0x7f032800b040).stop 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 shutdown_connections 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032002ff10 0x7f03200323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 --2- 172.21.15.115:0/164942868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03380f51a0 0x7f03380fd120 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 >> 172.21.15.115:0/164942868 conn(0x7f03380f09a0 msgr2=0x7f033805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 shutdown_connections 2024-01-21T07:33:02.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:02.573+0000 7f033cca9700 1 -- 172.21.15.115:0/164942868 wait complete. 2024-01-21T07:33:02.577 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:01.845518+0000 mgr.smithi107.adejht (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:02.573105+0000 mon.smithi107 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.115:0/164942868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:03.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:03.965 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:04.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:03.845872+0000 mgr.smithi107.adejht (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 -- 172.21.15.115:0/1657835551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 msgr2=0x7f94540f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 --2- 172.21.15.115:0/1657835551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94540f6850 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9444009fb0 tx=0x7f9444009b30).stop 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 -- 172.21.15.115:0/1657835551 shutdown_connections 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 --2- 172.21.15.115:0/1657835551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94540f6850 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 -- 172.21.15.115:0/1657835551 >> 172.21.15.115:0/1657835551 conn(0x7f94540f1b90 msgr2=0x7f94540f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 -- 172.21.15.115:0/1657835551 shutdown_connections 2024-01-21T07:33:05.082 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.078+0000 7f9459c63700 1 -- 172.21.15.115:0/1657835551 wait complete. 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f9459c63700 1 Processor -- start 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f9459c63700 1 -- start start 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f9459c63700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f9459c63700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454106ec0 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f94537fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f94537fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36574/0 (socket says 172.21.15.115:36574) 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.079+0000 7f94537fe700 1 -- 172.21.15.115:0/3992789598 learned_addr learned my addr 172.21.15.115:0/3992789598 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f94537fe700 1 -- 172.21.15.115:0/3992789598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444005040 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f94537fe700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9444014000 tx=0x7f944400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94440095f0 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94541070c0 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9444007e70 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9444014ea0 con 0x7f94540f6430 2024-01-21T07:33:05.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.080+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9454107550 con 0x7f94540f6430 2024-01-21T07:33:05.084 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9444025070 con 0x7f94540f6430 2024-01-21T07:33:05.084 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f943bfff700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 0x7f943c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:05.084 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9444040990 con 0x7f94540f6430 2024-01-21T07:33:05.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9454045ad0 con 0x7f94540f6430 2024-01-21T07:33:05.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f9452ffd700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 0x7f943c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:05.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.081+0000 7f9452ffd700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 0x7f943c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f94480096a0 tx=0x7f9448006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:05.088 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.086+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9444026350 con 0x7f94540f6430 2024-01-21T07:33:05.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.291+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f945405eba0 con 0x7f94540f6430 2024-01-21T07:33:05.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.292+0000 7f943bfff700 1 -- 172.21.15.115:0/3992789598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9444012330 con 0x7f94540f6430 2024-01-21T07:33:05.294 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:05.295 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:05.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 msgr2=0x7f943c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:05.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 0x7f943c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f94480096a0 tx=0x7f9448006b40).stop 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 msgr2=0x7f94541069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9444014000 tx=0x7f944400b040).stop 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 shutdown_connections 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f943c02ff60 0x7f943c032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 --2- 172.21.15.115:0/3992789598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94540f6430 0x7f94541069d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 >> 172.21.15.115:0/3992789598 conn(0x7f94540f1b90 msgr2=0x7f94540f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 shutdown_connections 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:05.294+0000 7f9459c63700 1 -- 172.21.15.115:0/3992789598 wait complete. 2024-01-21T07:33:05.297 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:05.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:05.293568+0000 mon.smithi107 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.115:0/3992789598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:06.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:06.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:06.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:05.846269+0000 mgr.smithi107.adejht (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:07.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 -- 172.21.15.115:0/1073086849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 msgr2=0x7fbc500f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 --2- 172.21.15.115:0/1073086849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc500f90f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fbc44009fb0 tx=0x7fbc44009b30).stop 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 -- 172.21.15.115:0/1073086849 shutdown_connections 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 --2- 172.21.15.115:0/1073086849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc500f90f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 -- 172.21.15.115:0/1073086849 >> 172.21.15.115:0/1073086849 conn(0x7fbc500f0b70 msgr2=0x7fbc500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 -- 172.21.15.115:0/1073086849 shutdown_connections 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.882+0000 7fbc561a5700 1 -- 172.21.15.115:0/1073086849 wait complete. 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.883+0000 7fbc561a5700 1 Processor -- start 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.883+0000 7fbc561a5700 1 -- start start 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.883+0000 7fbc561a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:07.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.883+0000 7fbc561a5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc50106bd0 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.883+0000 7fbc4f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc4f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36592/0 (socket says 172.21.15.115:36592) 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc4f7fe700 1 -- 172.21.15.115:0/3229539164 learned_addr learned my addr 172.21.15.115:0/3229539164 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc4f7fe700 1 -- 172.21.15.115:0/3229539164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44005040 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc4f7fe700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbc44013510 tx=0x7fbc4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc44009320 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc50106dd0 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc44013f00 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc44014dc0 con 0x7fbc50060280 2024-01-21T07:33:07.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.884+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc50107260 con 0x7fbc50060280 2024-01-21T07:33:07.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.885+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc44014420 con 0x7fbc50060280 2024-01-21T07:33:07.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.885+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc50045ad0 con 0x7fbc50060280 2024-01-21T07:33:07.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.885+0000 7fbc37fff700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 0x7fbc38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:07.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.885+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc44041050 con 0x7fbc50060280 2024-01-21T07:33:07.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.885+0000 7fbc4effd700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 0x7fbc38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:07.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.886+0000 7fbc4effd700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 0x7fbc38032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbc400096a0 tx=0x7fbc40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:07.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:07.890+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc440095f0 con 0x7fbc50060280 2024-01-21T07:33:08.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.095+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc50108110 con 0x7fbc50060280 2024-01-21T07:33:08.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.096+0000 7fbc37fff700 1 -- 172.21.15.115:0/3229539164 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc4401cb10 con 0x7fbc50060280 2024-01-21T07:33:08.098 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:08.098 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:08.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 msgr2=0x7fbc38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:08.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 0x7fbc38032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbc400096a0 tx=0x7fbc40006b40).stop 2024-01-21T07:33:08.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 msgr2=0x7fbc501066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbc44013510 tx=0x7fbc4400b040).stop 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 shutdown_connections 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbc3802ff60 0x7fbc38032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 --2- 172.21.15.115:0/3229539164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc50060280 0x7fbc501066e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 >> 172.21.15.115:0/3229539164 conn(0x7fbc500f0b70 msgr2=0x7fbc500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 shutdown_connections 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:08.097+0000 7fbc561a5700 1 -- 172.21.15.115:0/3229539164 wait complete. 2024-01-21T07:33:08.100 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:07.846605+0000 mgr.smithi107.adejht (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:09.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:08 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:08.097228+0000 mon.smithi107 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.115:0/3229539164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:09.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:09.519 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:09.846970+0000 mgr.smithi107.adejht (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:09.948247+0000 mon.smithi107 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- 172.21.15.115:0/1887116492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 msgr2=0x7f33ec0f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1887116492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec0f4ac0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f33dc009fb0 tx=0x7f33dc009b30).stop 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- 172.21.15.115:0/1887116492 shutdown_connections 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1887116492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec0f4ac0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- 172.21.15.115:0/1887116492 >> 172.21.15.115:0/1887116492 conn(0x7f33ec0f0120 msgr2=0x7f33ec0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- 172.21.15.115:0/1887116492 shutdown_connections 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- 172.21.15.115:0/1887116492 wait complete. 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 Processor -- start 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.661+0000 7f33eb59e700 1 -- start start 2024-01-21T07:33:10.665 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33eb59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33eb59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec106bd0 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33ea59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:55772/0 (socket says 172.21.15.115:55772) 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33ea59c700 1 -- 172.21.15.115:0/1482633192 learned_addr learned my addr 172.21.15.115:0/1482633192 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33ea59c700 1 -- 172.21.15.115:0/1482633192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc005040 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33ea59c700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f33dc013510 tx=0x7f33dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc009320 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec106dd0 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33dc013f00 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc014dc0 con 0x7f33ec0f46a0 2024-01-21T07:33:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.662+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec107260 con 0x7f33ec0f46a0 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.663+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33dc0095f0 con 0x7f33ec0f46a0 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.663+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec045ad0 con 0x7f33ec0f46a0 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.663+0000 7f33daffd700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 0x7f33d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.663+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33dc041000 con 0x7f33ec0f46a0 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.664+0000 7f33e9d9b700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 0x7f33d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.664+0000 7f33e9d9b700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 0x7f33d4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:10.671 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.668+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33dc014420 con 0x7f33ec0f46a0 2024-01-21T07:33:10.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.873+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33ec05eb50 con 0x7f33ec0f46a0 2024-01-21T07:33:10.876 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.874+0000 7f33daffd700 1 -- 172.21.15.115:0/1482633192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33dc010230 con 0x7f33ec0f46a0 2024-01-21T07:33:10.876 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:10.877 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 msgr2=0x7f33d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 0x7f33d4032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).stop 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 msgr2=0x7f33ec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f33dc013510 tx=0x7f33dc00b040).stop 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 shutdown_connections 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33d402ff60 0x7f33d4032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 --2- 172.21.15.115:0/1482633192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33ec0f46a0 0x7f33ec1066e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 >> 172.21.15.115:0/1482633192 conn(0x7f33ec0f0120 msgr2=0x7f33ec0f0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 shutdown_connections 2024-01-21T07:33:10.878 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:10.875+0000 7f33eb59e700 1 -- 172.21.15.115:0/1482633192 wait complete. 2024-01-21T07:33:10.879 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:09.949680+0000 mon.smithi107 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:33:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:10.875140+0000 mon.smithi107 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.115:0/1482633192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:12.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:12.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:13.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:11.847319+0000 mgr.smithi107.adejht (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2185669816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 msgr2=0x7f4a740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2185669816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a740f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f4a6c009960 tx=0x7f4a6c004d10).stop 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2185669816 shutdown_connections 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2185669816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a740f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2185669816 >> 172.21.15.115:0/2185669816 conn(0x7f4a740f0b70 msgr2=0x7f4a740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2185669816 shutdown_connections 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2185669816 wait complete. 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.445+0000 7f4a79e1f700 1 Processor -- start 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a79e1f700 1 -- start start 2024-01-21T07:33:13.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a79e1f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:13.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a79e1f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74105b20 con 0x7f4a740f5410 2024-01-21T07:33:13.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:13.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a737fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:55796/0 (socket says 172.21.15.115:55796) 2024-01-21T07:33:13.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a737fe700 1 -- 172.21.15.115:0/2617740584 learned_addr learned my addr 172.21.15.115:0/2617740584 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:13.450 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.446+0000 7f4a737fe700 1 -- 172.21.15.115:0/2617740584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c009640 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a737fe700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4a6c004b70 tx=0x7f4a6c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c018070 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a6c00b870 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a74105d20 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c01c560 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.447+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a741061b0 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a6c013630 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a6bfff700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 0x7f4a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a74045ad0 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a6c0411c0 con 0x7f4a740f5410 2024-01-21T07:33:13.451 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a72ffd700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 0x7f4a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:13.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.448+0000 7f4a72ffd700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 0x7f4a60032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4a5c0096a0 tx=0x7f4a5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:13.454 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.452+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a6c004810 con 0x7f4a740f5410 2024-01-21T07:33:13.660 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.658+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a740faf50 con 0x7f4a740f5410 2024-01-21T07:33:13.661 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.660+0000 7f4a6bfff700 1 -- 172.21.15.115:0/2617740584 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a6c012380 con 0x7f4a740f5410 2024-01-21T07:33:13.661 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:13.661 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:13.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 msgr2=0x7f4a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:13.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 0x7f4a60032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4a5c0096a0 tx=0x7f4a5c006b40).stop 2024-01-21T07:33:13.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 msgr2=0x7f4a74105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:13.663 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4a6c004b70 tx=0x7f4a6c004d10).stop 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 shutdown_connections 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f4a6002ff60 0x7f4a60032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 --2- 172.21.15.115:0/2617740584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a740f5410 0x7f4a74105630 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 >> 172.21.15.115:0/2617740584 conn(0x7f4a740f0b70 msgr2=0x7f4a740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 shutdown_connections 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:13.661+0000 7f4a79e1f700 1 -- 172.21.15.115:0/2617740584 wait complete. 2024-01-21T07:33:13.664 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:14.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:13.660412+0000 mon.smithi107 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.115:0/2617740584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:15.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:13.847653+0000 mgr.smithi107.adejht (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:15.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:15.095 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 -- 172.21.15.115:0/3157572136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 msgr2=0x7fe8840f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 --2- 172.21.15.115:0/3157572136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe8840f4be0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fe878009fb0 tx=0x7fe878009b30).stop 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 -- 172.21.15.115:0/3157572136 shutdown_connections 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 --2- 172.21.15.115:0/3157572136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe8840f4be0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 -- 172.21.15.115:0/3157572136 >> 172.21.15.115:0/3157572136 conn(0x7fe8840f02e0 msgr2=0x7fe8840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 -- 172.21.15.115:0/3157572136 shutdown_connections 2024-01-21T07:33:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.222+0000 7fe8894af700 1 -- 172.21.15.115:0/3157572136 wait complete. 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe8894af700 1 Processor -- start 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe8894af700 1 -- start start 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe8894af700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe8894af700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe884106bf0 con 0x7fe8840f47c0 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe882ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe882ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:55812/0 (socket says 172.21.15.115:55812) 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.223+0000 7fe882ffd700 1 -- 172.21.15.115:0/3431388467 learned_addr learned my addr 172.21.15.115:0/3431388467 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe882ffd700 1 -- 172.21.15.115:0/3431388467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe878005040 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe882ffd700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe878013510 tx=0x7fe87800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe878009320 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe878013f00 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe878014d90 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe884106df0 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.224+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe884107280 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.225+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8780095f0 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.225+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe884045ad0 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.225+0000 7fe86b7fe700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 0x7fe86c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.225+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe878040ff0 con 0x7fe8840f47c0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.225+0000 7fe8827fc700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 0x7fe86c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:16.229 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.226+0000 7fe8827fc700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 0x7fe86c032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe8740096a0 tx=0x7fe874006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:16.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.230+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe87806d2b0 con 0x7fe8840f47c0 2024-01-21T07:33:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:15.848017+0000 mgr.smithi107.adejht (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:16.437 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.435+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe884108130 con 0x7fe8840f47c0 2024-01-21T07:33:16.438 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.436+0000 7fe86b7fe700 1 -- 172.21.15.115:0/3431388467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe878014420 con 0x7fe8840f47c0 2024-01-21T07:33:16.438 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:16.438 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.437+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 msgr2=0x7fe86c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.437+0000 7fe8894af700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 0x7fe86c032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe8740096a0 tx=0x7fe874006b40).stop 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.437+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 msgr2=0x7fe884106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.437+0000 7fe8894af700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe878013510 tx=0x7fe87800b040).stop 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.437+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 shutdown_connections 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.438+0000 7fe8894af700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe86c02ffb0 0x7fe86c032470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.438+0000 7fe8894af700 1 --2- 172.21.15.115:0/3431388467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8840f47c0 0x7fe884106700 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.438+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 >> 172.21.15.115:0/3431388467 conn(0x7fe8840f02e0 msgr2=0x7fe8840f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.438+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 shutdown_connections 2024-01-21T07:33:16.440 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:16.438+0000 7fe8894af700 1 -- 172.21.15.115:0/3431388467 wait complete. 2024-01-21T07:33:16.441 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:17.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:16.437137+0000 mon.smithi107 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.115:0/3431388467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:17.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:17.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:17.848358+0000 mgr.smithi107.adejht (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:19.020 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 -- 172.21.15.115:0/2527225645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 msgr2=0x7f34d40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/2527225645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d40f90f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f34c4009fb0 tx=0x7f34c4009b30).stop 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 -- 172.21.15.115:0/2527225645 shutdown_connections 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/2527225645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d40f90f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 -- 172.21.15.115:0/2527225645 >> 172.21.15.115:0/2527225645 conn(0x7f34d40f0b70 msgr2=0x7f34d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 -- 172.21.15.115:0/2527225645 shutdown_connections 2024-01-21T07:33:19.021 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.019+0000 7f34dc5d9700 1 -- 172.21.15.115:0/2527225645 wait complete. 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34dc5d9700 1 Processor -- start 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34dc5d9700 1 -- start start 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34dc5d9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34dc5d9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4106bd0 con 0x7f34d4060280 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34da375700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34da375700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:55826/0 (socket says 172.21.15.115:55826) 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34da375700 1 -- 172.21.15.115:0/416015789 learned_addr learned my addr 172.21.15.115:0/416015789 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:19.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34da375700 1 -- 172.21.15.115:0/416015789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4005040 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.020+0000 7f34da375700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f34c4013510 tx=0x7f34c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c4009320 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4106dd0 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34c4013f00 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c4014dc0 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d4107260 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34c40095f0 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.021+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4045ad0 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.022+0000 7f34caffd700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 0x7f34c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.022+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34c4041000 con 0x7f34d4060280 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.022+0000 7f34d9b74700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 0x7f34c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:19.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.022+0000 7f34d9b74700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 0x7f34c0032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f34d00096a0 tx=0x7f34d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:19.027 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.026+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34c4014420 con 0x7f34d4060280 2024-01-21T07:33:19.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.234+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34d4108110 con 0x7f34d4060280 2024-01-21T07:33:19.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.234+0000 7f34caffd700 1 -- 172.21.15.115:0/416015789 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f34c4010230 con 0x7f34d4060280 2024-01-21T07:33:19.236 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:19.236 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:19.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 msgr2=0x7f34c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:19.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 0x7f34c0032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f34d00096a0 tx=0x7f34d0006b40).stop 2024-01-21T07:33:19.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 msgr2=0x7f34d41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:19.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f34c4013510 tx=0x7f34c400b040).stop 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 shutdown_connections 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34c002ff60 0x7f34c0032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 --2- 172.21.15.115:0/416015789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34d4060280 0x7f34d41066e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 >> 172.21.15.115:0/416015789 conn(0x7f34d40f0b70 msgr2=0x7f34d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 shutdown_connections 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:19.236+0000 7f34dc5d9700 1 -- 172.21.15.115:0/416015789 wait complete. 2024-01-21T07:33:19.238 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:19.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:19.234951+0000 mon.smithi107 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.115:0/416015789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:20.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:19.848638+0000 mgr.smithi107.adejht (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:20.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:20.652 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 -- 172.21.15.115:0/3367420643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 msgr2=0x7f46d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3367420643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d00f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f46c0009fb0 tx=0x7f46c0009b30).stop 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 -- 172.21.15.115:0/3367420643 shutdown_connections 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3367420643 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d00f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 -- 172.21.15.115:0/3367420643 >> 172.21.15.115:0/3367420643 conn(0x7f46d00f0b70 msgr2=0x7f46d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.763+0000 7f46d82c2700 1 -- 172.21.15.115:0/3367420643 shutdown_connections 2024-01-21T07:33:21.766 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d82c2700 1 -- 172.21.15.115:0/3367420643 wait complete. 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d82c2700 1 Processor -- start 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d82c2700 1 -- start start 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d82c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d82c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0060240 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d605e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d605e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:40844/0 (socket says 172.21.15.115:40844) 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.764+0000 7f46d605e700 1 -- 172.21.15.115:0/3502510845 learned_addr learned my addr 172.21.15.115:0/3502510845 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46d605e700 1 -- 172.21.15.115:0/3502510845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0005040 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46d605e700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f46c0013510 tx=0x7f46c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c0009320 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d0060440 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46c0013f00 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c0014d90 con 0x7f46d00f5410 2024-01-21T07:33:21.767 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.765+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d0109ae0 con 0x7f46d00f5410 2024-01-21T07:33:21.768 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.766+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46c00095f0 con 0x7f46d00f5410 2024-01-21T07:33:21.768 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.766+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d0045ad0 con 0x7f46d00f5410 2024-01-21T07:33:21.769 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.766+0000 7f46c6ffd700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 0x7f46bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:21.769 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.766+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46c0041090 con 0x7f46d00f5410 2024-01-21T07:33:21.769 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.766+0000 7f46d585d700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 0x7f46bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:21.769 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.767+0000 7f46d585d700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 0x7f46bc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:21.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.771+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46c006d2b0 con 0x7f46d00f5410 2024-01-21T07:33:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.977+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46d00fae60 con 0x7f46d00f5410 2024-01-21T07:33:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.977+0000 7f46c6ffd700 1 -- 172.21.15.115:0/3502510845 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46c0014420 con 0x7f46d00f5410 2024-01-21T07:33:21.978 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:21.978 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:21.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 msgr2=0x7f46bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:21.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 0x7f46bc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).stop 2024-01-21T07:33:21.980 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 msgr2=0x7f46d0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f46c0013510 tx=0x7f46c000b040).stop 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 shutdown_connections 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f46bc02ffb0 0x7f46bc032470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 --2- 172.21.15.115:0/3502510845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46d00f5410 0x7f46d0109430 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 >> 172.21.15.115:0/3502510845 conn(0x7f46d00f0b70 msgr2=0x7f46d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 shutdown_connections 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:21.979+0000 7f46d82c2700 1 -- 172.21.15.115:0/3502510845 wait complete. 2024-01-21T07:33:21.981 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:23.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:21.848965+0000 mgr.smithi107.adejht (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:23.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:21.977642+0000 mon.smithi107 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.115:0/3502510845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:23.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:23.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 -- 172.21.15.115:0/1141315940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 msgr2=0x7f6c8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 --2- 172.21.15.115:0/1141315940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c0f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f6c74009fb0 tx=0x7f6c74009b30).stop 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 -- 172.21.15.115:0/1141315940 shutdown_connections 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 --2- 172.21.15.115:0/1141315940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c0f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 -- 172.21.15.115:0/1141315940 >> 172.21.15.115:0/1141315940 conn(0x7f6c8c0f0b70 msgr2=0x7f6c8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 -- 172.21.15.115:0/1141315940 shutdown_connections 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 -- 172.21.15.115:0/1141315940 wait complete. 2024-01-21T07:33:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.504+0000 7f6c926e3700 1 Processor -- start 2024-01-21T07:33:24.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c926e3700 1 -- start start 2024-01-21T07:33:24.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c926e3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:24.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c926e3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c060220 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c8bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:40860/0 (socket says 172.21.15.115:40860) 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c8bfff700 1 -- 172.21.15.115:0/126307621 learned_addr learned my addr 172.21.15.115:0/126307621 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.505+0000 7f6c8bfff700 1 -- 172.21.15.115:0/126307621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74005040 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c8bfff700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6c74013510 tx=0x7f6c7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74009320 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c74013f00 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74014d90 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c060420 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.506+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c109b00 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c740095f0 con 0x7f6c8c0f5410 2024-01-21T07:33:24.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c059100 con 0x7f6c8c0f5410 2024-01-21T07:33:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c88ff9700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 0x7f6c78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c74040f90 con 0x7f6c8c0f5410 2024-01-21T07:33:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c8b7fe700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 0x7f6c78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.507+0000 7f6c8b7fe700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 0x7f6c78032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6c7c0096a0 tx=0x7f6c7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:24.512 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.512+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c74014420 con 0x7f6c8c0f5410 2024-01-21T07:33:24.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.718+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c8c045ad0 con 0x7f6c8c0f5410 2024-01-21T07:33:24.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.719+0000 7f6c88ff9700 1 -- 172.21.15.115:0/126307621 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c7401d7f0 con 0x7f6c8c0f5410 2024-01-21T07:33:24.720 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:24.720 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.720+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 msgr2=0x7f6c78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.720+0000 7f6c926e3700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 0x7f6c78032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6c7c0096a0 tx=0x7f6c7c006b40).stop 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.720+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 msgr2=0x7f6c8c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.720+0000 7f6c926e3700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6c74013510 tx=0x7f6c7400b040).stop 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 shutdown_connections 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6c7802ffb0 0x7f6c78032470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 --2- 172.21.15.115:0/126307621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c8c0f5410 0x7f6c8c109450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 >> 172.21.15.115:0/126307621 conn(0x7f6c8c0f0b70 msgr2=0x7f6c8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 shutdown_connections 2024-01-21T07:33:24.723 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:24.721+0000 7f6c926e3700 1 -- 172.21.15.115:0/126307621 wait complete. 2024-01-21T07:33:24.723 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:23.849241+0000 mgr.smithi107.adejht (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:24.719197+0000 mon.smithi107 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.115:0/126307621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:26.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:26.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:25.849595+0000 mgr.smithi107.adejht (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 -- 172.21.15.115:0/764530656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 msgr2=0x7f1de80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 --2- 172.21.15.115:0/764530656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de80f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f1dd8009fb0 tx=0x7f1dd8009b30).stop 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 -- 172.21.15.115:0/764530656 shutdown_connections 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 --2- 172.21.15.115:0/764530656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de80f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 -- 172.21.15.115:0/764530656 >> 172.21.15.115:0/764530656 conn(0x7f1de80f0b70 msgr2=0x7f1de80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:27.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 -- 172.21.15.115:0/764530656 shutdown_connections 2024-01-21T07:33:27.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.266+0000 7f1df066d700 1 -- 172.21.15.115:0/764530656 wait complete. 2024-01-21T07:33:27.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1df066d700 1 Processor -- start 2024-01-21T07:33:27.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1df066d700 1 -- start start 2024-01-21T07:33:27.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1df066d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:27.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1df066d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8060240 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1dee409700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1dee409700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:40868/0 (socket says 172.21.15.115:40868) 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.267+0000 7f1dee409700 1 -- 172.21.15.115:0/2359669845 learned_addr learned my addr 172.21.15.115:0/2359669845 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1dee409700 1 -- 172.21.15.115:0/2359669845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8005040 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1dee409700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1dd8014000 tx=0x7f1dd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd80095f0 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dd8007e70 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de8060440 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd8014ea0 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.268+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de8109ae0 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.269+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1dd8014690 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.269+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8045ad0 con 0x7f1de80f5410 2024-01-21T07:33:27.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.269+0000 7f1ddeffd700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 0x7f1dd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:27.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.269+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1dd8040870 con 0x7f1de80f5410 2024-01-21T07:33:27.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.269+0000 7f1dedc08700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 0x7f1dd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:27.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.270+0000 7f1dedc08700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 0x7f1dd4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f1de40096a0 tx=0x7f1de4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:27.275 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.274+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dd800f300 con 0x7f1de80f5410 2024-01-21T07:33:27.479 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.478+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1de80fae60 con 0x7f1de80f5410 2024-01-21T07:33:27.479 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.478+0000 7f1ddeffd700 1 -- 172.21.15.115:0/2359669845 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1dd800f300 con 0x7f1de80f5410 2024-01-21T07:33:27.480 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:27.480 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 msgr2=0x7f1dd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 0x7f1dd4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f1de40096a0 tx=0x7f1de4006b40).stop 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 msgr2=0x7f1de8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1dd8014000 tx=0x7f1dd800b040).stop 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 shutdown_connections 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1dd402ff60 0x7f1dd4032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 --2- 172.21.15.115:0/2359669845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1de80f5410 0x7f1de8109430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:27.481 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 >> 172.21.15.115:0/2359669845 conn(0x7f1de80f0b70 msgr2=0x7f1de80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:27.482 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 shutdown_connections 2024-01-21T07:33:27.482 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:27.480+0000 7f1df066d700 1 -- 172.21.15.115:0/2359669845 wait complete. 2024-01-21T07:33:27.482 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:28.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:27.478764+0000 mon.smithi107 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.115:0/2359669845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:28.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:28.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:29.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:27.849972+0000 mgr.smithi107.adejht (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 -- 172.21.15.115:0/3327388649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 msgr2=0x7fe7b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/3327388649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe7a4009fb0 tx=0x7fe7a4009b30).stop 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 -- 172.21.15.115:0/3327388649 shutdown_connections 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/3327388649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 -- 172.21.15.115:0/3327388649 >> 172.21.15.115:0/3327388649 conn(0x7fe7b00f0b70 msgr2=0x7fe7b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:30.022 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 -- 172.21.15.115:0/3327388649 shutdown_connections 2024-01-21T07:33:30.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.020+0000 7fe7b71d0700 1 -- 172.21.15.115:0/3327388649 wait complete. 2024-01-21T07:33:30.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b71d0700 1 Processor -- start 2024-01-21T07:33:30.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b71d0700 1 -- start start 2024-01-21T07:33:30.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b71d0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:30.023 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b71d0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00fb820 con 0x7fe7b00f5410 2024-01-21T07:33:30.024 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b4f6c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b4f6c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54778/0 (socket says 172.21.15.115:54778) 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.021+0000 7fe7b4f6c700 1 -- 172.21.15.115:0/1850403969 learned_addr learned my addr 172.21.15.115:0/1850403969 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7b4f6c700 1 -- 172.21.15.115:0/1850403969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a4005040 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7b4f6c700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7a4013510 tx=0x7fe7a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a4009320 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7a4013f00 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b00fb9c0 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a4014d90 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.022+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b00fbe50 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7a4014420 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b0059100 con 0x7fe7b00f5410 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7ad7fa700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 0x7fe7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:30.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7a40410c0 con 0x7fe7b00f5410 2024-01-21T07:33:30.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7affff700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 0x7fe7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:30.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.023+0000 7fe7affff700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 0x7fe7a0032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe79c0096a0 tx=0x7fe79c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:30.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.028+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7a40095f0 con 0x7fe7b00f5410 2024-01-21T07:33:30.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.233+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7b00fae60 con 0x7fe7b00f5410 2024-01-21T07:33:30.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.234+0000 7fe7ad7fa700 1 -- 172.21.15.115:0/1850403969 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe7a401db70 con 0x7fe7b00f5410 2024-01-21T07:33:30.235 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 msgr2=0x7fe7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 0x7fe7a0032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe79c0096a0 tx=0x7fe79c006b40).stop 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 msgr2=0x7fe7b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7a4013510 tx=0x7fe7a400b040).stop 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 shutdown_connections 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe7a002ff60 0x7fe7a0032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 --2- 172.21.15.115:0/1850403969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7b00f5410 0x7fe7b00fd3a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 >> 172.21.15.115:0/1850403969 conn(0x7fe7b00f0b70 msgr2=0x7fe7b005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:30.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 shutdown_connections 2024-01-21T07:33:30.238 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:30.236+0000 7fe7b71d0700 1 -- 172.21.15.115:0/1850403969 wait complete. 2024-01-21T07:33:30.238 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:29.850274+0000 mgr.smithi107.adejht (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:31.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:30.234306+0000 mon.smithi107 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.115:0/1850403969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:31.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:31.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:32.771 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 -- 172.21.15.115:0/2290271308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 msgr2=0x7f9d840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2290271308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d840f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9d78009fb0 tx=0x7f9d78009b30).stop 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 -- 172.21.15.115:0/2290271308 shutdown_connections 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2290271308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d840f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 -- 172.21.15.115:0/2290271308 >> 172.21.15.115:0/2290271308 conn(0x7f9d840f0b70 msgr2=0x7f9d840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 -- 172.21.15.115:0/2290271308 shutdown_connections 2024-01-21T07:33:32.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.769+0000 7f9d8977a700 1 -- 172.21.15.115:0/2290271308 wait complete. 2024-01-21T07:33:32.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d8977a700 1 Processor -- start 2024-01-21T07:33:32.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d8977a700 1 -- start start 2024-01-21T07:33:32.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d8977a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:32.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d8977a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84105b20 con 0x7f9d840f5410 2024-01-21T07:33:32.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d82ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d82ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54786/0 (socket says 172.21.15.115:54786) 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d82ffd700 1 -- 172.21.15.115:0/2853069927 learned_addr learned my addr 172.21.15.115:0/2853069927 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.770+0000 7f9d82ffd700 1 -- 172.21.15.115:0/2853069927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78005040 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d82ffd700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78009320 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d84105d20 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d78013f00 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78014d90 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841061b0 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d780095f0 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.771+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d84045ad0 con 0x7f9d840f5410 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.772+0000 7f9d6b7fe700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.772+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d78040fe0 con 0x7f9d840f5410 2024-01-21T07:33:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.772+0000 7f9d827fc700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.772+0000 7f9d827fc700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 0x7f9d6c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9d740096a0 tx=0x7f9d74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:32.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.776+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d78014420 con 0x7f9d840f5410 2024-01-21T07:33:32.987 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.985+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d840fae60 con 0x7f9d840f5410 2024-01-21T07:33:32.988 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.986+0000 7f9d6b7fe700 1 -- 172.21.15.115:0/2853069927 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d78010230 con 0x7f9d840f5410 2024-01-21T07:33:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:32.988 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 msgr2=0x7f9d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 0x7f9d6c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f9d740096a0 tx=0x7f9d74006b40).stop 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 msgr2=0x7f9d84105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).stop 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 shutdown_connections 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9d6c02ff60 0x7f9d6c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 --2- 172.21.15.115:0/2853069927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d840f5410 0x7f9d84105630 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 >> 172.21.15.115:0/2853069927 conn(0x7f9d840f0b70 msgr2=0x7f9d840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 shutdown_connections 2024-01-21T07:33:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:32.987+0000 7f9d8977a700 1 -- 172.21.15.115:0/2853069927 wait complete. 2024-01-21T07:33:32.991 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:31.850506+0000 mgr.smithi107.adejht (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:34.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:32.987436+0000 mon.smithi107 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.115:0/2853069927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:34.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:34.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:35.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:33.850824+0000 mgr.smithi107.adejht (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:35.626 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.623+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/2107191372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 msgr2=0x7f0b880f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.623+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/2107191372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b880f8f20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0b78009fb0 tx=0x7f0b78009b30).stop 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/2107191372 shutdown_connections 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/2107191372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b880f8f20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/2107191372 >> 172.21.15.115:0/2107191372 conn(0x7f0b880f09a0 msgr2=0x7f0b880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/2107191372 shutdown_connections 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/2107191372 wait complete. 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 Processor -- start 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- start start 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.624+0000 7f0b8e7c1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881069e0 con 0x7f0b880601e0 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8d7bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8d7bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54812/0 (socket says 172.21.15.115:54812) 2024-01-21T07:33:35.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8d7bf700 1 -- 172.21.15.115:0/3744749635 learned_addr learned my addr 172.21.15.115:0/3744749635 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8d7bf700 1 -- 172.21.15.115:0/3744749635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78005040 con 0x7f0b880601e0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8d7bf700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0b78013510 tx=0x7f0b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b78009320 con 0x7f0b880601e0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88106be0 con 0x7f0b880601e0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b78013f00 con 0x7f0b880601e0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b78014d90 con 0x7f0b880601e0 2024-01-21T07:33:35.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.625+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88107070 con 0x7f0b880601e0 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.626+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b780263a0 con 0x7f0b880601e0 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.626+0000 7f0b7dffb700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 0x7f0b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.626+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b78041080 con 0x7f0b880601e0 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.626+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88045ad0 con 0x7f0b880601e0 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.627+0000 7f0b8cfbe700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 0x7f0b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:35.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.627+0000 7f0b8cfbe700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 0x7f0b74032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0b840096a0 tx=0x7f0b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:35.633 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.631+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b78012920 con 0x7f0b880601e0 2024-01-21T07:33:35.839 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.837+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b88107e30 con 0x7f0b880601e0 2024-01-21T07:33:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.838+0000 7f0b7dffb700 1 -- 172.21.15.115:0/3744749635 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0b7802b020 con 0x7f0b880601e0 2024-01-21T07:33:35.840 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:35.840 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 msgr2=0x7f0b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 0x7f0b74032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0b840096a0 tx=0x7f0b84006b40).stop 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 msgr2=0x7f0b881064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0b78013510 tx=0x7f0b7800b040).stop 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 shutdown_connections 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0b7402ff60 0x7f0b74032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 --2- 172.21.15.115:0/3744749635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b880601e0 0x7f0b881064f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 >> 172.21.15.115:0/3744749635 conn(0x7f0b880f09a0 msgr2=0x7f0b880f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 shutdown_connections 2024-01-21T07:33:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:35.839+0000 7f0b8e7c1700 1 -- 172.21.15.115:0/3744749635 wait complete. 2024-01-21T07:33:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:35.839166+0000 mon.smithi107 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.115:0/3744749635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:37.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:37.216 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:37.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:35.851128+0000 mgr.smithi107.adejht (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:38.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 -- 172.21.15.115:0/3417116441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 msgr2=0x7f84080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:38.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 --2- 172.21.15.115:0/3417116441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84080f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f83f0009fb0 tx=0x7f83f0009b30).stop 2024-01-21T07:33:38.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 -- 172.21.15.115:0/3417116441 shutdown_connections 2024-01-21T07:33:38.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 --2- 172.21.15.115:0/3417116441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84080f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 -- 172.21.15.115:0/3417116441 >> 172.21.15.115:0/3417116441 conn(0x7f84080f0b70 msgr2=0x7f84080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 -- 172.21.15.115:0/3417116441 shutdown_connections 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.366+0000 7f840d87f700 1 -- 172.21.15.115:0/3417116441 wait complete. 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f840d87f700 1 Processor -- start 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f840d87f700 1 -- start start 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f840d87f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f840d87f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408060280 con 0x7f84080f5410 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f8406ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f8406ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:54840/0 (socket says 172.21.15.115:54840) 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.367+0000 7f8406ffd700 1 -- 172.21.15.115:0/242524538 learned_addr learned my addr 172.21.15.115:0/242524538 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f8406ffd700 1 -- 172.21.15.115:0/242524538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f0005040 con 0x7f84080f5410 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f8406ffd700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f83f0013510 tx=0x7f83f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83f0009320 con 0x7f84080f5410 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8408060480 con 0x7f84080f5410 2024-01-21T07:33:38.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8408109b20 con 0x7f84080f5410 2024-01-21T07:33:38.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83f0013f00 con 0x7f84080f5410 2024-01-21T07:33:38.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.368+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83f0014d60 con 0x7f84080f5410 2024-01-21T07:33:38.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8408045ad0 con 0x7f84080f5410 2024-01-21T07:33:38.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83f00095f0 con 0x7f84080f5410 2024-01-21T07:33:38.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f83ff7fe700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 0x7f83f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:38.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83f0040ff0 con 0x7f84080f5410 2024-01-21T07:33:38.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f84067fc700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 0x7f83f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:38.373 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.369+0000 7f84067fc700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 0x7f83f4032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f83f80096a0 tx=0x7f83f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:38.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.374+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83f0014420 con 0x7f84080f5410 2024-01-21T07:33:38.583 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.581+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84080fae60 con 0x7f84080f5410 2024-01-21T07:33:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.582+0000 7f83ff7fe700 1 -- 172.21.15.115:0/242524538 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f83f0012980 con 0x7f84080f5410 2024-01-21T07:33:38.584 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:38.584 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 msgr2=0x7f83f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 0x7f83f4032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f83f80096a0 tx=0x7f83f8006b40).stop 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 msgr2=0x7f84081093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f83f0013510 tx=0x7f83f000b040).stop 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 shutdown_connections 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f83f402ff60 0x7f83f4032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 --2- 172.21.15.115:0/242524538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84080f5410 0x7f84081093f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 >> 172.21.15.115:0/242524538 conn(0x7f84080f0b70 msgr2=0x7f84080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 shutdown_connections 2024-01-21T07:33:38.586 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:38.583+0000 7f840d87f700 1 -- 172.21.15.115:0/242524538 wait complete. 2024-01-21T07:33:38.587 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:39.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:37.851478+0000 mgr.smithi107.adejht (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:39.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:38.583270+0000 mon.smithi107 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.115:0/242524538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:39.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:39.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 -- 172.21.15.115:0/3051232534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 msgr2=0x7fc2540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 --2- 172.21.15.115:0/3051232534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc2540f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fc248009fb0 tx=0x7fc248009b30).stop 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 -- 172.21.15.115:0/3051232534 shutdown_connections 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 --2- 172.21.15.115:0/3051232534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc2540f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 -- 172.21.15.115:0/3051232534 >> 172.21.15.115:0/3051232534 conn(0x7fc2540f0b70 msgr2=0x7fc2540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:41.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 -- 172.21.15.115:0/3051232534 shutdown_connections 2024-01-21T07:33:41.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.113+0000 7fc2598c9700 1 -- 172.21.15.115:0/3051232534 wait complete. 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc2598c9700 1 Processor -- start 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc2598c9700 1 -- start start 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc2598c9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc2598c9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc254106c10 con 0x7fc2540f5410 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc252ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc252ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51752/0 (socket says 172.21.15.115:51752) 2024-01-21T07:33:41.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc252ffd700 1 -- 172.21.15.115:0/1408320690 learned_addr learned my addr 172.21.15.115:0/1408320690 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc252ffd700 1 -- 172.21.15.115:0/1408320690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248005040 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.114+0000 7fc252ffd700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc248013510 tx=0x7fc24800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc248009320 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc254106e10 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc248013f00 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc248014d90 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2541072a0 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc248014420 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.115+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc254045ad0 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.116+0000 7fc23b7fe700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 0x7fc23c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.116+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc248041000 con 0x7fc2540f5410 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.116+0000 7fc2527fc700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 0x7fc23c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:41.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.116+0000 7fc2527fc700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 0x7fc23c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc2440096a0 tx=0x7fc244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:41.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.120+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2480095f0 con 0x7fc2540f5410 2024-01-21T07:33:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:39.851768+0000 mgr.smithi107.adejht (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:41.326 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.324+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc254108150 con 0x7fc2540f5410 2024-01-21T07:33:41.326 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.324+0000 7fc23b7fe700 1 -- 172.21.15.115:0/1408320690 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc24801db70 con 0x7fc2540f5410 2024-01-21T07:33:41.327 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:41.327 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:41.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 msgr2=0x7fc23c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:41.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 0x7fc23c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc2440096a0 tx=0x7fc244006b40).stop 2024-01-21T07:33:41.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 msgr2=0x7fc254106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc248013510 tx=0x7fc24800b040).stop 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 shutdown_connections 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc23c02ff60 0x7fc23c032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 --2- 172.21.15.115:0/1408320690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2540f5410 0x7fc254106720 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 >> 172.21.15.115:0/1408320690 conn(0x7fc2540f0b70 msgr2=0x7fc25405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 shutdown_connections 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:41.326+0000 7fc2598c9700 1 -- 172.21.15.115:0/1408320690 wait complete. 2024-01-21T07:33:41.329 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:42.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:41.325821+0000 mon.smithi107 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.115:0/1408320690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:42.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:42.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:41.852088+0000 mgr.smithi107.adejht (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:43.882 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- 172.21.15.115:0/4039030248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 msgr2=0x7fc4500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 --2- 172.21.15.115:0/4039030248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fc444009fb0 tx=0x7fc444009b30).stop 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- 172.21.15.115:0/4039030248 shutdown_connections 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 --2- 172.21.15.115:0/4039030248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- 172.21.15.115:0/4039030248 >> 172.21.15.115:0/4039030248 conn(0x7fc4500f0b70 msgr2=0x7fc4500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- 172.21.15.115:0/4039030248 shutdown_connections 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- 172.21.15.115:0/4039030248 wait complete. 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 Processor -- start 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.879+0000 7fc455e7e700 1 -- start start 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc455e7e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc455e7e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500fb820 con 0x7fc4500f5410 2024-01-21T07:33:43.883 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc44f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc44f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51782/0 (socket says 172.21.15.115:51782) 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc44f7fe700 1 -- 172.21.15.115:0/2838496786 learned_addr learned my addr 172.21.15.115:0/2838496786 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc44f7fe700 1 -- 172.21.15.115:0/2838496786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444005040 con 0x7fc4500f5410 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc44f7fe700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fc444013510 tx=0x7fc44400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc444009320 con 0x7fc4500f5410 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4500fb9c0 con 0x7fc4500f5410 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc444013f00 con 0x7fc4500f5410 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc444014d90 con 0x7fc4500f5410 2024-01-21T07:33:43.884 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.880+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4500fbe50 con 0x7fc4500f5410 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.881+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4440263a0 con 0x7fc4500f5410 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.881+0000 7fc437fff700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 0x7fc438032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.881+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4440410e0 con 0x7fc4500f5410 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.881+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc450059100 con 0x7fc4500f5410 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.882+0000 7fc44effd700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 0x7fc438032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:43.885 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.882+0000 7fc44effd700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 0x7fc438032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc4400096a0 tx=0x7fc440006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:43.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:43.886+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc444012920 con 0x7fc4500f5410 2024-01-21T07:33:44.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.092+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4500fae60 con 0x7fc4500f5410 2024-01-21T07:33:44.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.093+0000 7fc437fff700 1 -- 172.21.15.115:0/2838496786 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc44402b020 con 0x7fc4500f5410 2024-01-21T07:33:44.095 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:44.095 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 msgr2=0x7fc438032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 0x7fc438032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc4400096a0 tx=0x7fc440006b40).stop 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 msgr2=0x7fc4500fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fc444013510 tx=0x7fc44400b040).stop 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 shutdown_connections 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc43802ff60 0x7fc438032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 --2- 172.21.15.115:0/2838496786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4500f5410 0x7fc4500fd3a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 >> 172.21.15.115:0/2838496786 conn(0x7fc4500f0b70 msgr2=0x7fc45005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:44.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 shutdown_connections 2024-01-21T07:33:44.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:44.095+0000 7fc455e7e700 1 -- 172.21.15.115:0/2838496786 wait complete. 2024-01-21T07:33:44.098 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:45.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:43.852451+0000 mgr.smithi107.adejht (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:45.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:44.094485+0000 mon.smithi107 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.115:0/2838496786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:45.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:45.493 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:46.643 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.639+0000 7f2631452700 1 -- 172.21.15.115:0/3851738703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 msgr2=0x7f262c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.639+0000 7f2631452700 1 --2- 172.21.15.115:0/3851738703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c0f6930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f261c009fb0 tx=0x7f261c009b30).stop 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- 172.21.15.115:0/3851738703 shutdown_connections 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 --2- 172.21.15.115:0/3851738703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c0f6930 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- 172.21.15.115:0/3851738703 >> 172.21.15.115:0/3851738703 conn(0x7f262c0f1c70 msgr2=0x7f262c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- 172.21.15.115:0/3851738703 shutdown_connections 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- 172.21.15.115:0/3851738703 wait complete. 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 Processor -- start 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- start start 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:46.644 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.640+0000 7f2631452700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c106c20 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f262affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f262affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:51798/0 (socket says 172.21.15.115:51798) 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f262affd700 1 -- 172.21.15.115:0/1932700317 learned_addr learned my addr 172.21.15.115:0/1932700317 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f262affd700 1 -- 172.21.15.115:0/1932700317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c005040 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f262affd700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f261c013510 tx=0x7f261c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f261c009320 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c106e20 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f261c013f00 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f261c014d90 con 0x7f262c0f6510 2024-01-21T07:33:46.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.641+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c1072b0 con 0x7f262c0f6510 2024-01-21T07:33:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.642+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f261c014420 con 0x7f262c0f6510 2024-01-21T07:33:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.642+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c045ad0 con 0x7f262c0f6510 2024-01-21T07:33:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.642+0000 7f26237fe700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 0x7f2618032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:46.646 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.642+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f261c0410c0 con 0x7f262c0f6510 2024-01-21T07:33:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.643+0000 7f262a7fc700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 0x7f2618032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:46.647 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.643+0000 7f262a7fc700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 0x7f2618032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f26140096a0 tx=0x7f2614006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:46.649 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.647+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f261c0095f0 con 0x7f262c0f6510 2024-01-21T07:33:46.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.852+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f262c0fbf60 con 0x7f262c0f6510 2024-01-21T07:33:46.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.852+0000 7f26237fe700 1 -- 172.21.15.115:0/1932700317 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f261c01db70 con 0x7f262c0f6510 2024-01-21T07:33:46.854 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:46.855 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:46.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 msgr2=0x7f2618032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:46.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 0x7f2618032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f26140096a0 tx=0x7f2614006b40).stop 2024-01-21T07:33:46.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 msgr2=0x7f262c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f261c013510 tx=0x7f261c00b040).stop 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 shutdown_connections 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f261802ffb0 0x7f2618032470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 --2- 172.21.15.115:0/1932700317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 >> 172.21.15.115:0/1932700317 conn(0x7f262c0f1c70 msgr2=0x7f262c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 shutdown_connections 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:46.854+0000 7f2631452700 1 -- 172.21.15.115:0/1932700317 wait complete. 2024-01-21T07:33:46.857 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:45.852723+0000 mgr.smithi107.adejht (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:46.853712+0000 mon.smithi107 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.115:0/1932700317' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:48.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:48.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:47.852952+0000 mgr.smithi107.adejht (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:49.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 -- 172.21.15.115:0/4263730236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a140f4740 msgr2=0x7f7a140f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:49.396 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 --2- 172.21.15.115:0/4263730236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a140f4740 0x7f7a140f4b60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f7a08009fb0 tx=0x7f7a08009b30).stop 2024-01-21T07:33:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 -- 172.21.15.115:0/4263730236 shutdown_connections 2024-01-21T07:33:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 --2- 172.21.15.115:0/4263730236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a140f4740 0x7f7a140f4b60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 -- 172.21.15.115:0/4263730236 >> 172.21.15.115:0/4263730236 conn(0x7f7a140f0260 msgr2=0x7f7a140f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 -- 172.21.15.115:0/4263730236 shutdown_connections 2024-01-21T07:33:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.394+0000 7f7a19320700 1 -- 172.21.15.115:0/4263730236 wait complete. 2024-01-21T07:33:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a19320700 1 Processor -- start 2024-01-21T07:33:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a19320700 1 -- start start 2024-01-21T07:33:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a19320700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a19320700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14107100 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a12ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a12ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45058/0 (socket says 172.21.15.115:45058) 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a12ffd700 1 -- 172.21.15.115:0/375993551 learned_addr learned my addr 172.21.15.115:0/375993551 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a12ffd700 1 -- 172.21.15.115:0/375993551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08005040 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.395+0000 7f7a12ffd700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7a08009f80 tx=0x7f7a08009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a080136c0 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a08013e70 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a14107300 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a08014420 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a14107e50 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a08025060 con 0x7f7a141067f0 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.396+0000 7f79fb7fe700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.397+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a08040ec0 con 0x7f7a141067f0 2024-01-21T07:33:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.397+0000 7f7a127fc700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.397+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a14059100 con 0x7f7a141067f0 2024-01-21T07:33:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.397+0000 7f7a127fc700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 0x7f79fc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7a040096a0 tx=0x7f7a04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:49.404 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.402+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a08014580 con 0x7f7a141067f0 2024-01-21T07:33:49.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.609+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a141084b0 con 0x7f7a141067f0 2024-01-21T07:33:49.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.610+0000 7f79fb7fe700 1 -- 172.21.15.115:0/375993551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a0801cb30 con 0x7f7a141067f0 2024-01-21T07:33:49.612 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:49.612 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 msgr2=0x7f79fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 0x7f79fc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7a040096a0 tx=0x7f7a04006b40).stop 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 msgr2=0x7f7a14106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7a08009f80 tx=0x7f7a08009b30).stop 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 shutdown_connections 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 --2- 172.21.15.115:0/375993551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a141067f0 0x7f7a14106c10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 >> 172.21.15.115:0/375993551 conn(0x7f7a140f0260 msgr2=0x7f7a140f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 shutdown_connections 2024-01-21T07:33:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:49.612+0000 7f7a19320700 1 -- 172.21.15.115:0/375993551 wait complete. 2024-01-21T07:33:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:50.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:49.611408+0000 mon.smithi107 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.115:0/375993551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:51.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:51.017 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:51.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:49.853297+0000 mgr.smithi107.adejht (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:52.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 -- 172.21.15.115:0/4249728115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 msgr2=0x7ff3e00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:52.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/4249728115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e00f90f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff3d0009960 tx=0x7ff3d0004d10).stop 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 -- 172.21.15.115:0/4249728115 shutdown_connections 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/4249728115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e00f90f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 -- 172.21.15.115:0/4249728115 >> 172.21.15.115:0/4249728115 conn(0x7ff3e00f0b70 msgr2=0x7ff3e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 -- 172.21.15.115:0/4249728115 shutdown_connections 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.119+0000 7ff3e5eda700 1 -- 172.21.15.115:0/4249728115 wait complete. 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3e5eda700 1 Processor -- start 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3e5eda700 1 -- start start 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3e5eda700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3e5eda700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e0106bd0 con 0x7ff3e0060280 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3df7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45082/0 (socket says 172.21.15.115:45082) 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.120+0000 7ff3df7fe700 1 -- 172.21.15.115:0/404088844 learned_addr learned my addr 172.21.15.115:0/404088844 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:52.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3df7fe700 1 -- 172.21.15.115:0/404088844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0009640 con 0x7ff3e0060280 2024-01-21T07:33:52.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3df7fe700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff3d0007bd0 tx=0x7ff3d0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d0018070 con 0x7ff3e0060280 2024-01-21T07:33:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0106dd0 con 0x7ff3e0060280 2024-01-21T07:33:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3d000b870 con 0x7ff3e0060280 2024-01-21T07:33:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d001d420 con 0x7ff3e0060280 2024-01-21T07:33:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.121+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e0107260 con 0x7ff3e0060280 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.122+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3d0024060 con 0x7ff3e0060280 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.122+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0045ad0 con 0x7ff3e0060280 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.122+0000 7ff3c7fff700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.122+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3d0041190 con 0x7ff3e0060280 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.122+0000 7ff3deffd700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.123+0000 7ff3deffd700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 0x7ff3c8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff3d40096a0 tx=0x7ff3d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:52.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.127+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3d0011790 con 0x7ff3e0060280 2024-01-21T07:33:52.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.334+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3e0108070 con 0x7ff3e0060280 2024-01-21T07:33:52.337 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.335+0000 7ff3c7fff700 1 -- 172.21.15.115:0/404088844 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3d0011790 con 0x7ff3e0060280 2024-01-21T07:33:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:52.337 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 msgr2=0x7ff3c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 0x7ff3c8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff3d40096a0 tx=0x7ff3d4006b40).stop 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 msgr2=0x7ff3e01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff3d0007bd0 tx=0x7ff3d0004d10).stop 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 shutdown_connections 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff3c802ff60 0x7ff3c8032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 --2- 172.21.15.115:0/404088844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3e0060280 0x7ff3e01066e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.336+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 >> 172.21.15.115:0/404088844 conn(0x7ff3e00f0b70 msgr2=0x7ff3e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.337+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 shutdown_connections 2024-01-21T07:33:52.339 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:52.337+0000 7ff3e5eda700 1 -- 172.21.15.115:0/404088844 wait complete. 2024-01-21T07:33:52.340 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:53.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:51.853593+0000 mgr.smithi107.adejht (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:53.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:52.336396+0000 mon.smithi107 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.115:0/404088844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:53.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:53.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:53.853903+0000 mgr.smithi107.adejht (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- 172.21.15.115:0/3077715691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 msgr2=0x7f47a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 --2- 172.21.15.115:0/3077715691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a80f55c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f479c009fb0 tx=0x7f479c009b30).stop 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- 172.21.15.115:0/3077715691 shutdown_connections 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 --2- 172.21.15.115:0/3077715691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a80f55c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- 172.21.15.115:0/3077715691 >> 172.21.15.115:0/3077715691 conn(0x7f47a80f09a0 msgr2=0x7f47a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- 172.21.15.115:0/3077715691 shutdown_connections 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- 172.21.15.115:0/3077715691 wait complete. 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 Processor -- start 2024-01-21T07:33:54.857 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.853+0000 7f47a759e700 1 -- start start 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a759e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a81058a0 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45094/0 (socket says 172.21.15.115:45094) 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a659c700 1 -- 172.21.15.115:0/644981761 learned_addr learned my addr 172.21.15.115:0/644981761 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a659c700 1 -- 172.21.15.115:0/644981761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c005040 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a659c700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f479c009830 tx=0x7f479c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479c009320 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a8105aa0 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f479c0136c0 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479c01ee90 con 0x7f47a80f51a0 2024-01-21T07:33:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.854+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a8105f30 con 0x7f47a80f51a0 2024-01-21T07:33:54.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.855+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f479c014440 con 0x7f47a80f51a0 2024-01-21T07:33:54.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.855+0000 7f4796ffd700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 0x7f47900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:54.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.855+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f479c041000 con 0x7f47a80f51a0 2024-01-21T07:33:54.859 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.855+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8045ad0 con 0x7f47a80f51a0 2024-01-21T07:33:54.860 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.856+0000 7f47a5d9b700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 0x7f47900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:54.860 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.856+0000 7f47a5d9b700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 0x7f47900323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f47980096a0 tx=0x7f4798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:54.862 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:54.860+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f479c0273f0 con 0x7f47a80f51a0 2024-01-21T07:33:55.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.065+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a80fabe0 con 0x7f47a80f51a0 2024-01-21T07:33:55.068 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.066+0000 7f4796ffd700 1 -- 172.21.15.115:0/644981761 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f47a80fabe0 con 0x7f47a80f51a0 2024-01-21T07:33:55.068 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:55.069 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.067+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 msgr2=0x7f47900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.067+0000 7f47a759e700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 0x7f47900323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f47980096a0 tx=0x7f4798006b40).stop 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.067+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 msgr2=0x7f47a81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.067+0000 7f47a759e700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f479c009830 tx=0x7f479c00b040).stop 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 shutdown_connections 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f479002ff10 0x7f47900323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 --2- 172.21.15.115:0/644981761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a80f51a0 0x7f47a81053b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 >> 172.21.15.115:0/644981761 conn(0x7f47a80f09a0 msgr2=0x7f47a80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 shutdown_connections 2024-01-21T07:33:55.070 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:55.068+0000 7f47a759e700 1 -- 172.21.15.115:0/644981761 wait complete. 2024-01-21T07:33:55.071 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:55.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:55.067579+0000 mon.smithi107 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.115:0/644981761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:56.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:56.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:33:56.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:56 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:55.854269+0000 mgr.smithi107.adejht (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:57.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- 172.21.15.115:0/3924357980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 msgr2=0x7f23000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:57.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 --2- 172.21.15.115:0/3924357980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f23000f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f22f0009fb0 tx=0x7f22f0009b30).stop 2024-01-21T07:33:57.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- 172.21.15.115:0/3924357980 shutdown_connections 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 --2- 172.21.15.115:0/3924357980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f23000f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- 172.21.15.115:0/3924357980 >> 172.21.15.115:0/3924357980 conn(0x7f23000f0b70 msgr2=0x7f23000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- 172.21.15.115:0/3924357980 shutdown_connections 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- 172.21.15.115:0/3924357980 wait complete. 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 Processor -- start 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.544+0000 7f230524a700 1 -- start start 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f230524a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f230524a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300105b20 con 0x7f23000f5410 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f22fed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f22fed9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45100/0 (socket says 172.21.15.115:45100) 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f22fed9d700 1 -- 172.21.15.115:0/3430202493 learned_addr learned my addr 172.21.15.115:0/3430202493 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.545+0000 7f22fed9d700 1 -- 172.21.15.115:0/3430202493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0005040 con 0x7f23000f5410 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f22fed9d700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f22f0013510 tx=0x7f22f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:57.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f0009320 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2300105d20 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22f0013f00 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f0014d90 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23001061b0 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.546+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22f00263a0 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.547+0000 7f22e77fe700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 0x7f22e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.547+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22f0040ff0 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.547+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2300045ad0 con 0x7f23000f5410 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.547+0000 7f22fe59c700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 0x7f22e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:33:57.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.547+0000 7f22fe59c700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 0x7f22e8032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f22f40096a0 tx=0x7f22f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:33:57.554 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.552+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22f0012920 con 0x7f23000f5410 2024-01-21T07:33:57.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.759+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23000faf50 con 0x7f23000f5410 2024-01-21T07:33:57.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.760+0000 7f22e77fe700 1 -- 172.21.15.115:0/3430202493 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22f002b020 con 0x7f23000f5410 2024-01-21T07:33:57.763 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:33:57.763 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 msgr2=0x7f22e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 0x7f22e8032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f22f40096a0 tx=0x7f22f4006b40).stop 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 msgr2=0x7f2300105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f22f0013510 tx=0x7f22f000b040).stop 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 shutdown_connections 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f22e802ff60 0x7f22e8032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:57.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 --2- 172.21.15.115:0/3430202493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23000f5410 0x7f2300105630 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:33:57.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 >> 172.21.15.115:0/3430202493 conn(0x7f23000f0b70 msgr2=0x7f23000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:33:57.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 shutdown_connections 2024-01-21T07:33:57.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:33:57.762+0000 7f230524a700 1 -- 172.21.15.115:0/3430202493 wait complete. 2024-01-21T07:33:57.765 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:33:58.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:33:57.761621+0000 mon.smithi107 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.115:0/3430202493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:33:59.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:57.854584+0000 mgr.smithi107.adejht (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:33:59.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:33:59.153 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.266+0000 7f37ac48c700 1 -- 172.21.15.115:0/4084369347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 msgr2=0x7f37a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.266+0000 7f37ac48c700 1 --2- 172.21.15.115:0/4084369347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a40f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f3794009fb0 tx=0x7f3794009b30).stop 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 -- 172.21.15.115:0/4084369347 shutdown_connections 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 --2- 172.21.15.115:0/4084369347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a40f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 -- 172.21.15.115:0/4084369347 >> 172.21.15.115:0/4084369347 conn(0x7f37a40f0b70 msgr2=0x7f37a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 -- 172.21.15.115:0/4084369347 shutdown_connections 2024-01-21T07:34:00.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 -- 172.21.15.115:0/4084369347 wait complete. 2024-01-21T07:34:00.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 Processor -- start 2024-01-21T07:34:00.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.267+0000 7f37ac48c700 1 -- start start 2024-01-21T07:34:00.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37ac48c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37ac48c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4060240 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37aa228700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37aa228700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45726/0 (socket says 172.21.15.115:45726) 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37aa228700 1 -- 172.21.15.115:0/1528355445 learned_addr learned my addr 172.21.15.115:0/1528355445 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37aa228700 1 -- 172.21.15.115:0/1528355445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794005040 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37aa228700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f3794013510 tx=0x7f379400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3794009320 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a4060440 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3794013f00 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3794014d90 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.268+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a4109ae0 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.269+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3794014420 con 0x7f37a40f5410 2024-01-21T07:34:00.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.269+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a4045ad0 con 0x7f37a40f5410 2024-01-21T07:34:00.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.269+0000 7f379affd700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 0x7f3790032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:00.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.269+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3794041080 con 0x7f37a40f5410 2024-01-21T07:34:00.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.269+0000 7f37a9a27700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 0x7f3790032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:00.273 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.270+0000 7f37a9a27700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 0x7f3790032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f37a00096a0 tx=0x7f37a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:00.276 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.274+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37940095f0 con 0x7f37a40f5410 2024-01-21T07:34:00.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:33:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:33:59.854894+0000 mgr.smithi107.adejht (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:00.483 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.481+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37a40fae60 con 0x7f37a40f5410 2024-01-21T07:34:00.484 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.481+0000 7f379affd700 1 -- 172.21.15.115:0/1528355445 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f379401db70 con 0x7f37a40f5410 2024-01-21T07:34:00.484 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:00.485 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 msgr2=0x7f3790032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:00.485 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 0x7f3790032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f37a00096a0 tx=0x7f37a0006b40).stop 2024-01-21T07:34:00.485 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 msgr2=0x7f37a4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f3794013510 tx=0x7f379400b040).stop 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 shutdown_connections 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f379002ff60 0x7f3790032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 --2- 172.21.15.115:0/1528355445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 >> 172.21.15.115:0/1528355445 conn(0x7f37a40f0b70 msgr2=0x7f37a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 shutdown_connections 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:00.483+0000 7f37ac48c700 1 -- 172.21.15.115:0/1528355445 wait complete. 2024-01-21T07:34:00.486 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:01.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:00.483012+0000 mon.smithi107 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.115:0/1528355445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:01.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:01.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 -- 172.21.15.115:0/3529949907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 msgr2=0x7feb640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 --2- 172.21.15.115:0/3529949907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb640f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7feb58009fb0 tx=0x7feb58009b30).stop 2024-01-21T07:34:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 -- 172.21.15.115:0/3529949907 shutdown_connections 2024-01-21T07:34:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 --2- 172.21.15.115:0/3529949907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb640f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 -- 172.21.15.115:0/3529949907 >> 172.21.15.115:0/3529949907 conn(0x7feb640f1c70 msgr2=0x7feb640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 -- 172.21.15.115:0/3529949907 shutdown_connections 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.954+0000 7feb69f04700 1 -- 172.21.15.115:0/3529949907 wait complete. 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.955+0000 7feb69f04700 1 Processor -- start 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.955+0000 7feb69f04700 1 -- start start 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.955+0000 7feb69f04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.955+0000 7feb69f04700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb64106c20 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.955+0000 7feb637fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb637fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45756/0 (socket says 172.21.15.115:45756) 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb637fe700 1 -- 172.21.15.115:0/3219945642 learned_addr learned my addr 172.21.15.115:0/3219945642 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb637fe700 1 -- 172.21.15.115:0/3219945642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58005040 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb637fe700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7feb58014000 tx=0x7feb5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb580095f0 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb64106e20 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb58007e70 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb58014ea0 con 0x7feb640f6510 2024-01-21T07:34:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.956+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb641072b0 con 0x7feb640f6510 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.957+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb58025070 con 0x7feb640f6510 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.957+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64045ad0 con 0x7feb640f6510 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.957+0000 7feb4bfff700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 0x7feb4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.957+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb580409e0 con 0x7feb640f6510 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.957+0000 7feb62ffd700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 0x7feb4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.958+0000 7feb62ffd700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 0x7feb4c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7feb540096a0 tx=0x7feb54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:02.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:02.962+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb5800f6c0 con 0x7feb640f6510 2024-01-21T07:34:03.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:01.855143+0000 mgr.smithi107.adejht (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.168+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb640fbf60 con 0x7feb640f6510 2024-01-21T07:34:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.169+0000 7feb4bfff700 1 -- 172.21.15.115:0/3219945642 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb5800f300 con 0x7feb640f6510 2024-01-21T07:34:03.171 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:03.172 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.170+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 msgr2=0x7feb4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.170+0000 7feb69f04700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 0x7feb4c032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7feb540096a0 tx=0x7feb54006b40).stop 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.170+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 msgr2=0x7feb64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.170+0000 7feb69f04700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7feb58014000 tx=0x7feb5800b040).stop 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 shutdown_connections 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7feb4c02ff60 0x7feb4c032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 --2- 172.21.15.115:0/3219945642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb640f6510 0x7feb64106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 >> 172.21.15.115:0/3219945642 conn(0x7feb640f1c70 msgr2=0x7feb640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 shutdown_connections 2024-01-21T07:34:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:03.171+0000 7feb69f04700 1 -- 172.21.15.115:0/3219945642 wait complete. 2024-01-21T07:34:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:03.169938+0000 mon.smithi107 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.115:0/3219945642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:04.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:04.553 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:03.855431+0000 mgr.smithi107.adejht (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:05.710 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 -- 172.21.15.115:0/3407546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 msgr2=0x7f0a2c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:05.710 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/3407546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f7d00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f0a20009fb0 tx=0x7f0a20009b30).stop 2024-01-21T07:34:05.710 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 -- 172.21.15.115:0/3407546 shutdown_connections 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/3407546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f7d00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 -- 172.21.15.115:0/3407546 >> 172.21.15.115:0/3407546 conn(0x7f0a2c0f0120 msgr2=0x7f0a2c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 -- 172.21.15.115:0/3407546 shutdown_connections 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.707+0000 7f0a2bfff700 1 -- 172.21.15.115:0/3407546 wait complete. 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2bfff700 1 Processor -- start 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2bfff700 1 -- start start 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2bfff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c0f5630 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45780/0 (socket says 172.21.15.115:45780) 2024-01-21T07:34:05.711 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.708+0000 7f0a2affd700 1 -- 172.21.15.115:0/675920247 learned_addr learned my addr 172.21.15.115:0/675920247 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a2affd700 1 -- 172.21.15.115:0/675920247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20005040 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a2affd700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0a20013510 tx=0x7f0a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a20009320 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a20013f00 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c0f57d0 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a20014dc0 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.709+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c0f5c60 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c059100 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.713 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a200095f0 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.714 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a137fe700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 0x7f0a14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:05.714 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a20041040 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.714 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a2a7fc700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 0x7f0a14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:05.714 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.710+0000 7f0a2a7fc700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 0x7f0a14032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f0a1c0096a0 tx=0x7f0a1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:05.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.715+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a20014420 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.921 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.920+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a2c0fbce0 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.921 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.920+0000 7f0a137fe700 1 -- 172.21.15.115:0/675920247 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0a20010230 con 0x7f0a2c0f78e0 2024-01-21T07:34:05.922 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:05.922 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 msgr2=0x7f0a14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 0x7f0a14032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f0a1c0096a0 tx=0x7f0a1c006b40).stop 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 msgr2=0x7f0a2c0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f0a20013510 tx=0x7f0a2000b040).stop 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 shutdown_connections 2024-01-21T07:34:05.923 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0a1402ff60 0x7f0a14032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:05.924 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 --2- 172.21.15.115:0/675920247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a2c0f78e0 0x7f0a2c0f71b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:05.924 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 >> 172.21.15.115:0/675920247 conn(0x7f0a2c0f0120 msgr2=0x7f0a2c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:05.924 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 shutdown_connections 2024-01-21T07:34:05.924 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:05.921+0000 7f0a2bfff700 1 -- 172.21.15.115:0/675920247 wait complete. 2024-01-21T07:34:05.924 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:05.855741+0000 mgr.smithi107.adejht (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:05.920764+0000 mon.smithi107 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.115:0/675920247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:07.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:07.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:07.856050+0000 mgr.smithi107.adejht (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:08.494 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 -- 172.21.15.115:0/3771451922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 msgr2=0x7f53f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:08.494 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/3771451922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f80f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f53ec009fb0 tx=0x7f53ec009b30).stop 2024-01-21T07:34:08.494 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 -- 172.21.15.115:0/3771451922 shutdown_connections 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/3771451922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f80f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 -- 172.21.15.115:0/3771451922 >> 172.21.15.115:0/3771451922 conn(0x7f53f80f1c70 msgr2=0x7f53f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 -- 172.21.15.115:0/3771451922 shutdown_connections 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.491+0000 7f53fdb6c700 1 -- 172.21.15.115:0/3771451922 wait complete. 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53fdb6c700 1 Processor -- start 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53fdb6c700 1 -- start start 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53fdb6c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53fdb6c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8069e20 con 0x7f53f80f6510 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53f77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53f77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:45792/0 (socket says 172.21.15.115:45792) 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.492+0000 7f53f77fe700 1 -- 172.21.15.115:0/808849871 learned_addr learned my addr 172.21.15.115:0/808849871 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53f77fe700 1 -- 172.21.15.115:0/808849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec005040 con 0x7f53f80f6510 2024-01-21T07:34:08.495 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53f77fe700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f53ec014000 tx=0x7f53ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:08.496 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec0095f0 con 0x7f53f80f6510 2024-01-21T07:34:08.496 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f8069fc0 con 0x7f53f80f6510 2024-01-21T07:34:08.496 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53ec007e70 con 0x7f53f80f6510 2024-01-21T07:34:08.496 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec014ea0 con 0x7f53f80f6510 2024-01-21T07:34:08.496 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.493+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f806a450 con 0x7f53f80f6510 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.494+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53ec025070 con 0x7f53f80f6510 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.494+0000 7f53dffff700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 0x7f53e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.494+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53ec040a40 con 0x7f53f80f6510 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.494+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8059100 con 0x7f53f80f6510 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.494+0000 7f53f6ffd700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 0x7f53e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:08.497 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.495+0000 7f53f6ffd700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 0x7f53e0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f53e80096a0 tx=0x7f53e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:08.500 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.499+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53ec026350 con 0x7f53f80f6510 2024-01-21T07:34:08.707 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.706+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53f8045ad0 con 0x7f53f80f6510 2024-01-21T07:34:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.707+0000 7f53dffff700 1 -- 172.21.15.115:0/808849871 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53ec02d090 con 0x7f53f80f6510 2024-01-21T07:34:08.762 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:08.762 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 msgr2=0x7f53e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 0x7f53e0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f53e80096a0 tx=0x7f53e8006b40).stop 2024-01-21T07:34:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 msgr2=0x7f53f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f53ec014000 tx=0x7f53ec00b040).stop 2024-01-21T07:34:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 shutdown_connections 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e002ff60 0x7f53e0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 --2- 172.21.15.115:0/808849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f80f6510 0x7f53f806b9a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 >> 172.21.15.115:0/808849871 conn(0x7f53f80f1c70 msgr2=0x7f53f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 shutdown_connections 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:08.708+0000 7f53fdb6c700 1 -- 172.21.15.115:0/808849871 wait complete. 2024-01-21T07:34:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:09.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:08 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:08.707153+0000 mon.smithi107 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.115:0/808849871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:10.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:10.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:09.856345+0000 mgr.smithi107.adejht (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:09.949141+0000 mon.smithi107 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:34:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:09.950480+0000 mon.smithi107 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 -- 172.21.15.115:0/376926310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 msgr2=0x7fb0bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 --2- 172.21.15.115:0/376926310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc0f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb0ac009fb0 tx=0x7fb0ac009b30).stop 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 -- 172.21.15.115:0/376926310 shutdown_connections 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 --2- 172.21.15.115:0/376926310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc0f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 -- 172.21.15.115:0/376926310 >> 172.21.15.115:0/376926310 conn(0x7fb0bc0f1c70 msgr2=0x7fb0bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 -- 172.21.15.115:0/376926310 shutdown_connections 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 -- 172.21.15.115:0/376926310 wait complete. 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.207+0000 7fb0c184d700 1 Processor -- start 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0c184d700 1 -- start start 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0c184d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:11.209 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0c184d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc106c20 con 0x7fb0bc0f6510 2024-01-21T07:34:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0baffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0baffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35388/0 (socket says 172.21.15.115:35388) 2024-01-21T07:34:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0baffd700 1 -- 172.21.15.115:0/286277541 learned_addr learned my addr 172.21.15.115:0/286277541 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0baffd700 1 -- 172.21.15.115:0/286277541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac005040 con 0x7fb0bc0f6510 2024-01-21T07:34:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.208+0000 7fb0baffd700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb0ac013510 tx=0x7fb0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0ac009320 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0ac013f00 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc106e20 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0ac014d90 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc1072b0 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0ac0095f0 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.209+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc045ad0 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.210+0000 7fb0a37fe700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 0x7fb0a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.210+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0ac0410c0 con 0x7fb0bc0f6510 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.210+0000 7fb0ba7fc700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 0x7fb0a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.210+0000 7fb0ba7fc700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 0x7fb0a4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb0b00096a0 tx=0x7fb0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:11.215 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.214+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0ac014420 con 0x7fb0bc0f6510 2024-01-21T07:34:11.423 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.422+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0bc0fbf60 con 0x7fb0bc0f6510 2024-01-21T07:34:11.424 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.423+0000 7fb0a37fe700 1 -- 172.21.15.115:0/286277541 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb0ac01d7f0 con 0x7fb0bc0f6510 2024-01-21T07:34:11.424 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:11.424 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:11.425 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 msgr2=0x7fb0a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 0x7fb0a4032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb0b00096a0 tx=0x7fb0b0006b40).stop 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 msgr2=0x7fb0bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb0ac013510 tx=0x7fb0ac00b040).stop 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 shutdown_connections 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb0a402ff60 0x7fb0a4032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 --2- 172.21.15.115:0/286277541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0bc0f6510 0x7fb0bc106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 >> 172.21.15.115:0/286277541 conn(0x7fb0bc0f1c70 msgr2=0x7fb0bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 shutdown_connections 2024-01-21T07:34:11.426 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:11.424+0000 7fb0c184d700 1 -- 172.21.15.115:0/286277541 wait complete. 2024-01-21T07:34:11.427 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:11.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:11 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:11.423329+0000 mon.smithi107 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.115:0/286277541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:12.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:12.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:13.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:11.856665+0000 mgr.smithi107.adejht (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:13.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 -- 172.21.15.115:0/2174816720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 msgr2=0x7faf4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:13.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 --2- 172.21.15.115:0/2174816720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c0f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7faf48009fb0 tx=0x7faf48009b30).stop 2024-01-21T07:34:13.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 -- 172.21.15.115:0/2174816720 shutdown_connections 2024-01-21T07:34:13.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 --2- 172.21.15.115:0/2174816720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c0f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:13.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 -- 172.21.15.115:0/2174816720 >> 172.21.15.115:0/2174816720 conn(0x7faf4c0f1c70 msgr2=0x7faf4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:13.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 -- 172.21.15.115:0/2174816720 shutdown_connections 2024-01-21T07:34:13.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.923+0000 7faf53310700 1 -- 172.21.15.115:0/2174816720 wait complete. 2024-01-21T07:34:13.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf53310700 1 Processor -- start 2024-01-21T07:34:13.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf53310700 1 -- start start 2024-01-21T07:34:13.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf53310700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:13.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf53310700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c106c20 con 0x7faf4c0f6510 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf510ac700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf510ac700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35408/0 (socket says 172.21.15.115:35408) 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf510ac700 1 -- 172.21.15.115:0/4148413438 learned_addr learned my addr 172.21.15.115:0/4148413438 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf510ac700 1 -- 172.21.15.115:0/4148413438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48005040 con 0x7faf4c0f6510 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.924+0000 7faf510ac700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7faf48014000 tx=0x7faf4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:13.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.925+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf480095f0 con 0x7faf4c0f6510 2024-01-21T07:34:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.925+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c106e20 con 0x7faf4c0f6510 2024-01-21T07:34:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.925+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf48007e70 con 0x7faf4c0f6510 2024-01-21T07:34:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.925+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf48014ea0 con 0x7faf4c0f6510 2024-01-21T07:34:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.925+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c1072b0 con 0x7faf4c0f6510 2024-01-21T07:34:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf48025070 con 0x7faf4c0f6510 2024-01-21T07:34:13.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c045ad0 con 0x7faf4c0f6510 2024-01-21T07:34:13.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf41ffb700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 0x7faf38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:13.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf48040940 con 0x7faf4c0f6510 2024-01-21T07:34:13.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf508ab700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 0x7faf38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:13.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.926+0000 7faf508ab700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 0x7faf38032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:13.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:13.931+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf480079c0 con 0x7faf4c0f6510 2024-01-21T07:34:14.136 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.135+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf4c0fbf60 con 0x7faf4c0f6510 2024-01-21T07:34:14.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.136+0000 7faf41ffb700 1 -- 172.21.15.115:0/4148413438 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf4800f300 con 0x7faf4c0f6510 2024-01-21T07:34:14.137 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:14.137 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:14.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 msgr2=0x7faf38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:14.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 0x7faf38032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).stop 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 msgr2=0x7faf4c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7faf48014000 tx=0x7faf4800b040).stop 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 shutdown_connections 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7faf3802ff60 0x7faf38032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 --2- 172.21.15.115:0/4148413438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c0f6510 0x7faf4c106730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 >> 172.21.15.115:0/4148413438 conn(0x7faf4c0f1c70 msgr2=0x7faf4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 shutdown_connections 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:14.137+0000 7faf53310700 1 -- 172.21.15.115:0/4148413438 wait complete. 2024-01-21T07:34:14.139 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:14.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:13.857007+0000 mgr.smithi107.adejht (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:15.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:14.136096+0000 mon.smithi107 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.115:0/4148413438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:15.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:15.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:15.857376+0000 mgr.smithi107.adejht (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 -- 172.21.15.115:0/3626688861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 msgr2=0x7f35000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 --2- 172.21.15.115:0/3626688861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35000f90f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f34e8009fb0 tx=0x7f34e8009b30).stop 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 -- 172.21.15.115:0/3626688861 shutdown_connections 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 --2- 172.21.15.115:0/3626688861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35000f90f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 -- 172.21.15.115:0/3626688861 >> 172.21.15.115:0/3626688861 conn(0x7f35000f0b70 msgr2=0x7f35000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 -- 172.21.15.115:0/3626688861 shutdown_connections 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 -- 172.21.15.115:0/3626688861 wait complete. 2024-01-21T07:34:16.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.690+0000 7f3506741700 1 Processor -- start 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f3506741700 1 -- start start 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f3506741700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f3506741700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3500106bd0 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f34fffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f34fffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35428/0 (socket says 172.21.15.115:35428) 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.691+0000 7f34fffff700 1 -- 172.21.15.115:0/1586063918 learned_addr learned my addr 172.21.15.115:0/1586063918 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f34fffff700 1 -- 172.21.15.115:0/1586063918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8005040 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f34fffff700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f34e8013510 tx=0x7f34e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e8009320 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3500106dd0 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e8013f00 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e8014dc0 con 0x7f3500060280 2024-01-21T07:34:16.694 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.692+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3500107260 con 0x7f3500060280 2024-01-21T07:34:16.695 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.693+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34e8014420 con 0x7f3500060280 2024-01-21T07:34:16.695 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.693+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3500045ad0 con 0x7f3500060280 2024-01-21T07:34:16.696 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.693+0000 7f34fcff9700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 0x7f34ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:16.696 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.693+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34e80410e0 con 0x7f3500060280 2024-01-21T07:34:16.696 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.693+0000 7f34ff7fe700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 0x7f34ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:16.696 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.694+0000 7f34ff7fe700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 0x7f34ec032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f34f00096a0 tx=0x7f34f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:16.698 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.698+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e80095f0 con 0x7f3500060280 2024-01-21T07:34:16.905 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.903+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35001080b0 con 0x7f3500060280 2024-01-21T07:34:16.905 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.904+0000 7f34fcff9700 1 -- 172.21.15.115:0/1586063918 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f34e801cb10 con 0x7f3500060280 2024-01-21T07:34:16.906 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:16.906 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 msgr2=0x7f34ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 0x7f34ec032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f34f00096a0 tx=0x7f34f0006b40).stop 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 msgr2=0x7f35001066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f34e8013510 tx=0x7f34e800b040).stop 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 shutdown_connections 2024-01-21T07:34:16.907 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f34ec02ff60 0x7f34ec032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:16.908 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 --2- 172.21.15.115:0/1586063918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3500060280 0x7f35001066e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:16.908 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 >> 172.21.15.115:0/1586063918 conn(0x7f35000f0b70 msgr2=0x7f35000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:16.908 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 shutdown_connections 2024-01-21T07:34:16.908 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:16.906+0000 7f3506741700 1 -- 172.21.15.115:0/1586063918 wait complete. 2024-01-21T07:34:16.908 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:18.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:16.904620+0000 mon.smithi107 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.115:0/1586063918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:18.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:18.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:19.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:17.857715+0000 mgr.smithi107.adejht (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 -- 172.21.15.115:0/3472676841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 msgr2=0x7fe3480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 --2- 172.21.15.115:0/3472676841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe3480f55c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fe338009fb0 tx=0x7fe338009b30).stop 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 -- 172.21.15.115:0/3472676841 shutdown_connections 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 --2- 172.21.15.115:0/3472676841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe3480f55c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 -- 172.21.15.115:0/3472676841 >> 172.21.15.115:0/3472676841 conn(0x7fe3480f09a0 msgr2=0x7fe3480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 -- 172.21.15.115:0/3472676841 shutdown_connections 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 -- 172.21.15.115:0/3472676841 wait complete. 2024-01-21T07:34:19.474 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.471+0000 7fe34d4db700 1 Processor -- start 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe34d4db700 1 -- start start 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe34d4db700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe34d4db700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348060220 con 0x7fe3480f51a0 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe347fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe347fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47882/0 (socket says 172.21.15.115:47882) 2024-01-21T07:34:19.475 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe347fff700 1 -- 172.21.15.115:0/120729829 learned_addr learned my addr 172.21.15.115:0/120729829 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe347fff700 1 -- 172.21.15.115:0/120729829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338005040 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.472+0000 7fe347fff700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe338013510 tx=0x7fe33800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.473+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe338009320 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.473+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe348060420 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.473+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe338013f00 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.473+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe338014dc0 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.473+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3481098b0 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3380095f0 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe348059100 con 0x7fe3480f51a0 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe344ff9700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 0x7fe330032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:19.477 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe338041040 con 0x7fe3480f51a0 2024-01-21T07:34:19.478 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe3477fe700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 0x7fe330032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:19.478 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.474+0000 7fe3477fe700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 0x7fe330032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe33c0096a0 tx=0x7fe33c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:19.480 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.478+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe338014420 con 0x7fe3480f51a0 2024-01-21T07:34:19.690 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.688+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe348045ad0 con 0x7fe3480f51a0 2024-01-21T07:34:19.691 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.689+0000 7fe344ff9700 1 -- 172.21.15.115:0/120729829 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe338010230 con 0x7fe3480f51a0 2024-01-21T07:34:19.691 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:19.691 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:19.692 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 msgr2=0x7fe330032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:19.692 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 0x7fe330032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe33c0096a0 tx=0x7fe33c006b40).stop 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 msgr2=0x7fe348109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe338013510 tx=0x7fe33800b040).stop 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 shutdown_connections 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe33002ff60 0x7fe330032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 --2- 172.21.15.115:0/120729829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3480f51a0 0x7fe348109200 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 >> 172.21.15.115:0/120729829 conn(0x7fe3480f09a0 msgr2=0x7fe3480f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 shutdown_connections 2024-01-21T07:34:19.693 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:19.690+0000 7fe34d4db700 1 -- 172.21.15.115:0/120729829 wait complete. 2024-01-21T07:34:19.694 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:20.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:19.690106+0000 mon.smithi107 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.115:0/120729829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:21.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:21.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:21.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:19.858051+0000 mgr.smithi107.adejht (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 -- 172.21.15.115:0/2531838418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 msgr2=0x7f0c8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 --2- 172.21.15.115:0/2531838418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c0f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f0c80009fb0 tx=0x7f0c80009b30).stop 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 -- 172.21.15.115:0/2531838418 shutdown_connections 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 --2- 172.21.15.115:0/2531838418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c0f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 -- 172.21.15.115:0/2531838418 >> 172.21.15.115:0/2531838418 conn(0x7f0c8c0f0b70 msgr2=0x7f0c8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 -- 172.21.15.115:0/2531838418 shutdown_connections 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 -- 172.21.15.115:0/2531838418 wait complete. 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.229+0000 7f0c92de4700 1 Processor -- start 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c92de4700 1 -- start start 2024-01-21T07:34:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c92de4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:22.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c92de4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c106c10 con 0x7f0c8c0f5410 2024-01-21T07:34:22.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c90b80700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:22.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c90b80700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47910/0 (socket says 172.21.15.115:47910) 2024-01-21T07:34:22.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c90b80700 1 -- 172.21.15.115:0/4152278221 learned_addr learned my addr 172.21.15.115:0/4152278221 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:22.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.230+0000 7f0c90b80700 1 -- 172.21.15.115:0/4152278221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80005040 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c90b80700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0c80013510 tx=0x7f0c8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c80009320 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c106e10 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c80013f00 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c80014d90 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c1072a0 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.231+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c800095f0 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.232+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8c045ad0 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.232+0000 7f0c897fa700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 0x7f0c7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.232+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c800410a0 con 0x7f0c8c0f5410 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.232+0000 7f0c8bfff700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 0x7f0c7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:22.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.232+0000 7f0c8bfff700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 0x7f0c7c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:22.239 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.237+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c8001d7f0 con 0x7f0c8c0f5410 2024-01-21T07:34:22.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.447+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c8c108150 con 0x7f0c8c0f5410 2024-01-21T07:34:22.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.447+0000 7f0c897fa700 1 -- 172.21.15.115:0/4152278221 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c80010230 con 0x7f0c8c0f5410 2024-01-21T07:34:22.450 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:22.450 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 msgr2=0x7f0c7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 0x7f0c7c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).stop 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 msgr2=0x7f0c8c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0c80013510 tx=0x7f0c8000b040).stop 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 shutdown_connections 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f0c7c02ff60 0x7f0c7c032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 --2- 172.21.15.115:0/4152278221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c8c0f5410 0x7f0c8c106720 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 >> 172.21.15.115:0/4152278221 conn(0x7f0c8c0f0b70 msgr2=0x7f0c8c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 shutdown_connections 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:22.449+0000 7f0c92de4700 1 -- 172.21.15.115:0/4152278221 wait complete. 2024-01-21T07:34:22.452 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:21.858380+0000 mgr.smithi107.adejht (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:22.448758+0000 mon.smithi107 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.115:0/4152278221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:23.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:23.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:23.858699+0000 mgr.smithi107.adejht (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 -- 172.21.15.115:0/3968949413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 msgr2=0x7f61700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 --2- 172.21.15.115:0/3968949413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f61700f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f6160009fb0 tx=0x7f6160009b30).stop 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 -- 172.21.15.115:0/3968949413 shutdown_connections 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 --2- 172.21.15.115:0/3968949413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f61700f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 -- 172.21.15.115:0/3968949413 >> 172.21.15.115:0/3968949413 conn(0x7f61700f0b70 msgr2=0x7f61700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 -- 172.21.15.115:0/3968949413 shutdown_connections 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.962+0000 7f61779dc700 1 -- 172.21.15.115:0/3968949413 wait complete. 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f61779dc700 1 Processor -- start 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f61779dc700 1 -- start start 2024-01-21T07:34:24.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f61779dc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f61779dc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6170105b20 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f6175778700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f6175778700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47922/0 (socket says 172.21.15.115:47922) 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f6175778700 1 -- 172.21.15.115:0/4071559951 learned_addr learned my addr 172.21.15.115:0/4071559951 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.963+0000 7f6175778700 1 -- 172.21.15.115:0/4071559951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160005040 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f6175778700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f6160013510 tx=0x7f616000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6160009320 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6160013f00 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6170105d20 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6160014d90 con 0x7f61700f5410 2024-01-21T07:34:24.967 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.964+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61701061b0 con 0x7f61700f5410 2024-01-21T07:34:24.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.965+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61600263a0 con 0x7f61700f5410 2024-01-21T07:34:24.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.965+0000 7f6165ffb700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 0x7f615c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:24.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.965+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61600410c0 con 0x7f61700f5410 2024-01-21T07:34:24.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.965+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6170045ad0 con 0x7f61700f5410 2024-01-21T07:34:24.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.965+0000 7f6174f77700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 0x7f615c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:24.969 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.966+0000 7f6174f77700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 0x7f615c032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f616c0096a0 tx=0x7f616c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:24.972 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:24.970+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6160012920 con 0x7f61700f5410 2024-01-21T07:34:25.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.176+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61700faf50 con 0x7f61700f5410 2024-01-21T07:34:25.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.176+0000 7f6165ffb700 1 -- 172.21.15.115:0/4071559951 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f616002b020 con 0x7f61700f5410 2024-01-21T07:34:25.179 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:25.179 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:25.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 msgr2=0x7f615c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:25.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 0x7f615c032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f616c0096a0 tx=0x7f616c006b40).stop 2024-01-21T07:34:25.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 msgr2=0x7f6170105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f6160013510 tx=0x7f616000b040).stop 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 shutdown_connections 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f615c02ff60 0x7f615c032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 --2- 172.21.15.115:0/4071559951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61700f5410 0x7f6170105630 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 >> 172.21.15.115:0/4071559951 conn(0x7f61700f0b70 msgr2=0x7f61700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 shutdown_connections 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:25.178+0000 7f61779dc700 1 -- 172.21.15.115:0/4071559951 wait complete. 2024-01-21T07:34:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:25.566 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:25.177992+0000 mon.smithi107 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.115:0/4071559951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:26.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:26 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:25.859050+0000 mgr.smithi107.adejht (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:26.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:26.567 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:27.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 -- 172.21.15.115:0/3419322419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 msgr2=0x7f15140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:27.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 --2- 172.21.15.115:0/3419322419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f15140f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1504009fb0 tx=0x7f1504009b30).stop 2024-01-21T07:34:27.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 -- 172.21.15.115:0/3419322419 shutdown_connections 2024-01-21T07:34:27.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 --2- 172.21.15.115:0/3419322419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f15140f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:27.750 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 -- 172.21.15.115:0/3419322419 >> 172.21.15.115:0/3419322419 conn(0x7f15140f0b70 msgr2=0x7f15140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 -- 172.21.15.115:0/3419322419 shutdown_connections 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.746+0000 7f151b786700 1 -- 172.21.15.115:0/3419322419 wait complete. 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f151b786700 1 Processor -- start 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f151b786700 1 -- start start 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f151b786700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f151b786700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1514060220 con 0x7f15140f5410 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f1519522700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f1519522700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47932/0 (socket says 172.21.15.115:47932) 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f1519522700 1 -- 172.21.15.115:0/1210243340 learned_addr learned my addr 172.21.15.115:0/1210243340 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f1519522700 1 -- 172.21.15.115:0/1210243340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1504005040 con 0x7f15140f5410 2024-01-21T07:34:27.751 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.747+0000 7f1519522700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1504013510 tx=0x7f150400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:27.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.748+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1504009320 con 0x7f15140f5410 2024-01-21T07:34:27.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.748+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1514060420 con 0x7f15140f5410 2024-01-21T07:34:27.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.748+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1504013f00 con 0x7f15140f5410 2024-01-21T07:34:27.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.748+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1504014d90 con 0x7f15140f5410 2024-01-21T07:34:27.752 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.748+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1514109ad0 con 0x7f15140f5410 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15040095f0 con 0x7f15140f5410 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514059100 con 0x7f15140f5410 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f1509ffb700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1504041080 con 0x7f15140f5410 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f1518d21700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:27.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.749+0000 7f1518d21700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 0x7f1500032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f15100096a0 tx=0x7f1510006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:27.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.754+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1504014420 con 0x7f15140f5410 2024-01-21T07:34:27.960 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.958+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1514045ad0 con 0x7f15140f5410 2024-01-21T07:34:27.961 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.959+0000 7f1509ffb700 1 -- 172.21.15.115:0/1210243340 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1504010230 con 0x7f15140f5410 2024-01-21T07:34:27.961 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:27.961 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 msgr2=0x7f1500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 0x7f1500032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f15100096a0 tx=0x7f1510006b40).stop 2024-01-21T07:34:27.962 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 msgr2=0x7f1514109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1504013510 tx=0x7f150400b040).stop 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 shutdown_connections 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f150002ff60 0x7f1500032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 --2- 172.21.15.115:0/1210243340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15140f5410 0x7f1514109420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 >> 172.21.15.115:0/1210243340 conn(0x7f15140f0b70 msgr2=0x7f15140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 shutdown_connections 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:27.960+0000 7f151b786700 1 -- 172.21.15.115:0/1210243340 wait complete. 2024-01-21T07:34:27.963 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:27.859412+0000 mgr.smithi107.adejht (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:29.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:27.960172+0000 mon.smithi107 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.115:0/1210243340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:29.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:29.348 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:29.859673+0000 mgr.smithi107.adejht (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 -- 172.21.15.115:0/3690058274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 msgr2=0x7f8a840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 --2- 172.21.15.115:0/3690058274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f90f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8a78009fb0 tx=0x7f8a78009b30).stop 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 -- 172.21.15.115:0/3690058274 shutdown_connections 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 --2- 172.21.15.115:0/3690058274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f90f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 -- 172.21.15.115:0/3690058274 >> 172.21.15.115:0/3690058274 conn(0x7f8a840f0b70 msgr2=0x7f8a840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 -- 172.21.15.115:0/3690058274 shutdown_connections 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.509+0000 7f8a8b252700 1 -- 172.21.15.115:0/3690058274 wait complete. 2024-01-21T07:34:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a8b252700 1 Processor -- start 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a8b252700 1 -- start start 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a8b252700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a8b252700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a840f4ff0 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a88fee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a88fee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39924/0 (socket says 172.21.15.115:39924) 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.510+0000 7f8a88fee700 1 -- 172.21.15.115:0/4293626461 learned_addr learned my addr 172.21.15.115:0/4293626461 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a88fee700 1 -- 172.21.15.115:0/4293626461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78005040 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a88fee700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8a78009830 tx=0x7f8a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78009320 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a780136c0 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a840f5190 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a7801ee90 con 0x7f8a84060280 2024-01-21T07:34:30.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.511+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a840f5620 con 0x7f8a84060280 2024-01-21T07:34:30.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a7801e420 con 0x7f8a84060280 2024-01-21T07:34:30.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a84059100 con 0x7f8a84060280 2024-01-21T07:34:30.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a817fa700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:30.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a78040f10 con 0x7f8a84060280 2024-01-21T07:34:30.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a83fff700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:30.516 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.512+0000 7f8a83fff700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 0x7f8a740323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:30.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.517+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a780095f0 con 0x7f8a84060280 2024-01-21T07:34:30.726 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.724+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a840fbf50 con 0x7f8a84060280 2024-01-21T07:34:30.727 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.725+0000 7f8a817fa700 1 -- 172.21.15.115:0/4293626461 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a7802c2d0 con 0x7f8a84060280 2024-01-21T07:34:30.728 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:30.728 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 msgr2=0x7f8a740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 0x7f8a740323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 msgr2=0x7f8a840f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8a78009830 tx=0x7f8a7800b040).stop 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 shutdown_connections 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 --2- 172.21.15.115:0/4293626461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a84060280 0x7f8a840f6b70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 >> 172.21.15.115:0/4293626461 conn(0x7f8a840f0b70 msgr2=0x7f8a840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:30.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 shutdown_connections 2024-01-21T07:34:30.730 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:30.727+0000 7f8a8b252700 1 -- 172.21.15.115:0/4293626461 wait complete. 2024-01-21T07:34:30.730 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:31.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:30.726490+0000 mon.smithi107 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.115:0/4293626461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:32.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:32.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:33.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:31.860008+0000 mgr.smithi107.adejht (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:33.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:32.145763+0000 mon.smithi107 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi107", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T07:34:33.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:32.148242+0000 mon.smithi107 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi107", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T07:34:33.066 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:32.149032+0000 mon.smithi107 (mon.0) 235 : audit [DBG] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 -- 172.21.15.115:0/3675505127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 msgr2=0x7f1fc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 --2- 172.21.15.115:0/3675505127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc00f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f1fb4009fb0 tx=0x7f1fb4009b30).stop 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 -- 172.21.15.115:0/3675505127 shutdown_connections 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 --2- 172.21.15.115:0/3675505127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc00f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 -- 172.21.15.115:0/3675505127 >> 172.21.15.115:0/3675505127 conn(0x7f1fc00f1c70 msgr2=0x7f1fc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:33.257 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 -- 172.21.15.115:0/3675505127 shutdown_connections 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.252+0000 7f1fc6582700 1 -- 172.21.15.115:0/3675505127 wait complete. 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fc6582700 1 Processor -- start 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fc6582700 1 -- start start 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fc6582700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fc6582700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0069e20 con 0x7f1fc00f6510 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fbffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fbffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39936/0 (socket says 172.21.15.115:39936) 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.253+0000 7f1fbffff700 1 -- 172.21.15.115:0/1345712825 learned_addr learned my addr 172.21.15.115:0/1345712825 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fbffff700 1 -- 172.21.15.115:0/1345712825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fc00f6510 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fbffff700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1fb4014000 tx=0x7f1fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb40095f0 con 0x7f1fc00f6510 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fb4007e70 con 0x7f1fc00f6510 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4014ea0 con 0x7f1fc00f6510 2024-01-21T07:34:33.258 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc0069fc0 con 0x7f1fc00f6510 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.254+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc006a450 con 0x7f1fc00f6510 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fb4025070 con 0x7f1fc00f6510 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fbcff9700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 0x7f1fa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fb4040a60 con 0x7f1fc00f6510 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0059100 con 0x7f1fc00f6510 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fbf7fe700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 0x7f1fa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:33.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.255+0000 7f1fbf7fe700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 0x7f1fa8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:33.262 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.260+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fb40079c0 con 0x7f1fc00f6510 2024-01-21T07:34:33.468 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.466+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc006ab50 con 0x7f1fc00f6510 2024-01-21T07:34:33.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.467+0000 7f1fbcff9700 1 -- 172.21.15.115:0/1345712825 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1fb400f300 con 0x7f1fc00f6510 2024-01-21T07:34:33.469 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:33.469 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 msgr2=0x7f1fa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 0x7f1fa8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).stop 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 msgr2=0x7f1fc006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1fb4014000 tx=0x7f1fb400b040).stop 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 shutdown_connections 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1fa802ff60 0x7f1fa8032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 --2- 172.21.15.115:0/1345712825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00f6510 0x7f1fc006b9a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.468+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 >> 172.21.15.115:0/1345712825 conn(0x7f1fc00f1c70 msgr2=0x7f1fc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.469+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 shutdown_connections 2024-01-21T07:34:33.471 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:33.469+0000 7f1fc6582700 1 -- 172.21.15.115:0/1345712825 wait complete. 2024-01-21T07:34:33.472 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:34.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:34:32.144579+0000 mgr.smithi107.adejht (mgr.14162) 163 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi107 ... 2024-01-21T07:34:34.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T07:34:34.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:29:56Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:30:34Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:31:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:32:39Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:33:17Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:33:54Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: main() 2024-01-21T07:34:34.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: r = ctx.func(ctx) 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: return func(ctx) 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: raise RuntimeError('uid/gid not found') 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: RuntimeError: uid/gid not found 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: yield (conn, connr) 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T07:34:34.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: code, '\n'.join(err))) 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi107 ... 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:29:56Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:30:34Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:31:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi107 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:32:39Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:33:17Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr time="2024-01-21T07:33:54Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T07:34:34.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: main() 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: r = ctx.func(ctx) 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: return func(ctx) 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: raise RuntimeError('uid/gid not found') 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: RuntimeError: uid/gid not found 2024-01-21T07:34:34.320 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:34:32.149603+0000 mgr.smithi107.adejht (mgr.14162) 164 : cephadm [INF] Deploying daemon crash.smithi107 on smithi107 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.468345+0000 mon.smithi107 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.115:0/1345712825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.534624+0000 mon.smithi107 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.646828+0000 mon.smithi107 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.657625+0000 mon.smithi107 (mon.0) 239 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.658062+0000 mon.smithi107 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T07:34:34.321 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.661452+0000 mon.smithi107 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:34:34.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:34.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:35.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:33.658430+0000 mgr.smithi107.adejht (mgr.14162) 165 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T07:34:35.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:34:33.664281+0000 mgr.smithi107.adejht (mgr.14162) 166 : cephadm [INF] Deploying daemon grafana.smithi107 on smithi107 2024-01-21T07:34:35.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:33.860350+0000 mgr.smithi107.adejht (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:36.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 -- 172.21.15.115:0/3209753003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 msgr2=0x7f93540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:36.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 --2- 172.21.15.115:0/3209753003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f93540f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f9344009fb0 tx=0x7f9344009b30).stop 2024-01-21T07:34:36.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 -- 172.21.15.115:0/3209753003 shutdown_connections 2024-01-21T07:34:36.031 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 --2- 172.21.15.115:0/3209753003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f93540f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 -- 172.21.15.115:0/3209753003 >> 172.21.15.115:0/3209753003 conn(0x7f93540f0b70 msgr2=0x7f93540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.027+0000 7f935c521700 1 -- 172.21.15.115:0/3209753003 shutdown_connections 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935c521700 1 -- 172.21.15.115:0/3209753003 wait complete. 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935c521700 1 Processor -- start 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935c521700 1 -- start start 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935c521700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935c521700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9354105b20 con 0x7f93540f5410 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935a2bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935a2bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39958/0 (socket says 172.21.15.115:39958) 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.028+0000 7f935a2bd700 1 -- 172.21.15.115:0/1474530796 learned_addr learned my addr 172.21.15.115:0/1474530796 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f935a2bd700 1 -- 172.21.15.115:0/1474530796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344005040 con 0x7f93540f5410 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f935a2bd700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9344013510 tx=0x7f934400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:36.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344009320 con 0x7f93540f5410 2024-01-21T07:34:36.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9344013f00 con 0x7f93540f5410 2024-01-21T07:34:36.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9354105d20 con 0x7f93540f5410 2024-01-21T07:34:36.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9344014d90 con 0x7f93540f5410 2024-01-21T07:34:36.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.029+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93541061b0 con 0x7f93540f5410 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9344014420 con 0x7f93540f5410 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f934affd700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 0x7f9340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9354045ad0 con 0x7f93540f5410 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93440410c0 con 0x7f93540f5410 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f9359abc700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 0x7f9340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:36.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.030+0000 7f9359abc700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 0x7f9340032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f93500096a0 tx=0x7f9350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:36.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.035+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f934406d2b0 con 0x7f93540f5410 2024-01-21T07:34:36.243 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.241+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93540faf50 con 0x7f93540f5410 2024-01-21T07:34:36.244 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.242+0000 7f934affd700 1 -- 172.21.15.115:0/1474530796 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f934401dd50 con 0x7f93540f5410 2024-01-21T07:34:36.244 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:36.244 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:36.245 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 msgr2=0x7f9340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:36.245 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 0x7f9340032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f93500096a0 tx=0x7f9350006b40).stop 2024-01-21T07:34:36.245 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 msgr2=0x7f9354105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:36.245 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9344013510 tx=0x7f934400b040).stop 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 shutdown_connections 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f934002ff60 0x7f9340032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 --2- 172.21.15.115:0/1474530796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93540f5410 0x7f9354105630 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 >> 172.21.15.115:0/1474530796 conn(0x7f93540f0b70 msgr2=0x7f93540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 shutdown_connections 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:36.243+0000 7f935c521700 1 -- 172.21.15.115:0/1474530796 wait complete. 2024-01-21T07:34:36.246 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:35.860628+0000 mgr.smithi107.adejht (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:37.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:36.243129+0000 mon.smithi107 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.115:0/1474530796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:37.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:37.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:37.860895+0000 mgr.smithi107.adejht (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:38.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 -- 172.21.15.115:0/2130670130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 msgr2=0x7fb6b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:38.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 --2- 172.21.15.115:0/2130670130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b00f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb6a4009fb0 tx=0x7fb6a4009b30).stop 2024-01-21T07:34:38.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 -- 172.21.15.115:0/2130670130 shutdown_connections 2024-01-21T07:34:38.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 --2- 172.21.15.115:0/2130670130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b00f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:38.732 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 -- 172.21.15.115:0/2130670130 >> 172.21.15.115:0/2130670130 conn(0x7fb6b00f0b70 msgr2=0x7fb6b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 -- 172.21.15.115:0/2130670130 shutdown_connections 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.729+0000 7fb6b487b700 1 -- 172.21.15.115:0/2130670130 wait complete. 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6b487b700 1 Processor -- start 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6b487b700 1 -- start start 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6b487b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:38.733 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6b487b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b0060280 con 0x7fb6b00f5410 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39972/0 (socket says 172.21.15.115:39972) 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6ae59c700 1 -- 172.21.15.115:0/1055507157 learned_addr learned my addr 172.21.15.115:0/1055507157 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.730+0000 7fb6ae59c700 1 -- 172.21.15.115:0/1055507157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4005040 con 0x7fb6b00f5410 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb6ae59c700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb6a4013510 tx=0x7fb6a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a4009320 con 0x7fb6b00f5410 2024-01-21T07:34:38.735 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b0060480 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6a4013f00 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a4014d90 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b0109b20 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.731+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6a40095f0 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.732+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b0045ad0 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.732+0000 7fb69effd700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 0x7fb698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.732+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6a40410c0 con 0x7fb6b00f5410 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.732+0000 7fb6add9b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 0x7fb698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:38.736 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.732+0000 7fb6add9b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 0x7fb698032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6a00096a0 tx=0x7fb6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:38.738 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.736+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6a401d7f0 con 0x7fb6b00f5410 2024-01-21T07:34:38.946 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.944+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6b010a340 con 0x7fb6b00f5410 2024-01-21T07:34:38.947 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.945+0000 7fb69effd700 1 -- 172.21.15.115:0/1055507157 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6a4010230 con 0x7fb6b00f5410 2024-01-21T07:34:38.947 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:38.947 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.946+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 msgr2=0x7fb698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.946+0000 7fb6b487b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 0x7fb698032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6a00096a0 tx=0x7fb6a0006b40).stop 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.946+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 msgr2=0x7fb6b01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.946+0000 7fb6b487b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb6a4013510 tx=0x7fb6a400b040).stop 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 shutdown_connections 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fb69802ff60 0x7fb698032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 --2- 172.21.15.115:0/1055507157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6b00f5410 0x7fb6b01093f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 >> 172.21.15.115:0/1055507157 conn(0x7fb6b00f0b70 msgr2=0x7fb6b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 shutdown_connections 2024-01-21T07:34:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:38.947+0000 7fb6b487b700 1 -- 172.21.15.115:0/1055507157 wait complete. 2024-01-21T07:34:38.950 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:38.946483+0000 mon.smithi107 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.115:0/1055507157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:40.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:39.861192+0000 mgr.smithi107.adejht (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:40.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:40.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:41.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 -- 172.21.15.115:0/3078217970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 msgr2=0x7f814c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:41.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 --2- 172.21.15.115:0/3078217970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c0f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f8140009fb0 tx=0x7f8140009b30).stop 2024-01-21T07:34:41.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 -- 172.21.15.115:0/3078217970 shutdown_connections 2024-01-21T07:34:41.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 --2- 172.21.15.115:0/3078217970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c0f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:41.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 -- 172.21.15.115:0/3078217970 >> 172.21.15.115:0/3078217970 conn(0x7f814c0f0b70 msgr2=0x7f814c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:41.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 -- 172.21.15.115:0/3078217970 shutdown_connections 2024-01-21T07:34:41.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.520+0000 7f8150c4e700 1 -- 172.21.15.115:0/3078217970 wait complete. 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f8150c4e700 1 Processor -- start 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f8150c4e700 1 -- start start 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f8150c4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f8150c4e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c105b20 con 0x7f814c0f5410 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f814a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f814a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59956/0 (socket says 172.21.15.115:59956) 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f814a59c700 1 -- 172.21.15.115:0/971388255 learned_addr learned my addr 172.21.15.115:0/971388255 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f814a59c700 1 -- 172.21.15.115:0/971388255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8140005040 con 0x7f814c0f5410 2024-01-21T07:34:41.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.521+0000 7f814a59c700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8140013510 tx=0x7f814000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8140009320 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8140013f00 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814c105d20 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8140014d90 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814c1061b0 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8140014420 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.522+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814c045ad0 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.523+0000 7f813affd700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 0x7f8134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.523+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81400410c0 con 0x7f814c0f5410 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.523+0000 7f8149d9b700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 0x7f8134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.523+0000 7f8149d9b700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 0x7f8134032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f813c0096a0 tx=0x7f813c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:41.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.531+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81400095f0 con 0x7f814c0f5410 2024-01-21T07:34:41.740 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.737+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f814c0fae60 con 0x7f814c0f5410 2024-01-21T07:34:41.740 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.738+0000 7f813affd700 1 -- 172.21.15.115:0/971388255 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f814001db70 con 0x7f814c0f5410 2024-01-21T07:34:41.740 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:41.740 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:41.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 msgr2=0x7f8134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:41.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 0x7f8134032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f813c0096a0 tx=0x7f813c006b40).stop 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 msgr2=0x7f814c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8140013510 tx=0x7f814000b040).stop 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 shutdown_connections 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f813402ff60 0x7f8134032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 --2- 172.21.15.115:0/971388255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f814c0f5410 0x7f814c105630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 >> 172.21.15.115:0/971388255 conn(0x7f814c0f0b70 msgr2=0x7f814c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 shutdown_connections 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:41.739+0000 7f8150c4e700 1 -- 172.21.15.115:0/971388255 wait complete. 2024-01-21T07:34:41.742 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:42.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:41.739157+0000 mon.smithi107 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.115:0/971388255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:43.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:43.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:41.861458+0000 mgr.smithi107.adejht (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:44.189 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 -- 172.21.15.115:0/2631769338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 msgr2=0x7f5d980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:44.189 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/2631769338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d980f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5d88009fb0 tx=0x7f5d88009b30).stop 2024-01-21T07:34:44.189 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 -- 172.21.15.115:0/2631769338 shutdown_connections 2024-01-21T07:34:44.189 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/2631769338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d980f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 -- 172.21.15.115:0/2631769338 >> 172.21.15.115:0/2631769338 conn(0x7f5d980f0b70 msgr2=0x7f5d980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 -- 172.21.15.115:0/2631769338 shutdown_connections 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.185+0000 7f5d9e19e700 1 -- 172.21.15.115:0/2631769338 wait complete. 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d9e19e700 1 Processor -- start 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d9e19e700 1 -- start start 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d9e19e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d9e19e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98106c10 con 0x7f5d980f5410 2024-01-21T07:34:44.190 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d977fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59988/0 (socket says 172.21.15.115:59988) 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d977fe700 1 -- 172.21.15.115:0/629775455 learned_addr learned my addr 172.21.15.115:0/629775455 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.186+0000 7f5d977fe700 1 -- 172.21.15.115:0/629775455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d88005040 con 0x7f5d980f5410 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d977fe700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5d88013510 tx=0x7f5d8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d88009320 con 0x7f5d980f5410 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d88013f00 con 0x7f5d980f5410 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d98106e10 con 0x7f5d980f5410 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d88014d90 con 0x7f5d980f5410 2024-01-21T07:34:44.191 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.187+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d981072a0 con 0x7f5d980f5410 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d880095f0 con 0x7f5d980f5410 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d98045ad0 con 0x7f5d980f5410 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d7ffff700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 0x7f5d80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d88041000 con 0x7f5d980f5410 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d96ffd700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 0x7f5d80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:44.192 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.188+0000 7f5d96ffd700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 0x7f5d80032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5d8c0096a0 tx=0x7f5d8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:44.194 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.192+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d88014420 con 0x7f5d980f5410 2024-01-21T07:34:44.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:43.861698+0000 mgr.smithi107.adejht (mgr.14162) 172 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:44.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.399+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d98108150 con 0x7f5d980f5410 2024-01-21T07:34:44.401 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.399+0000 7f5d7ffff700 1 -- 172.21.15.115:0/629775455 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d8801d7f0 con 0x7f5d980f5410 2024-01-21T07:34:44.401 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:44.402 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 msgr2=0x7f5d80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 0x7f5d80032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5d8c0096a0 tx=0x7f5d8c006b40).stop 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 msgr2=0x7f5d98106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5d88013510 tx=0x7f5d8800b040).stop 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 shutdown_connections 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5d8002ff60 0x7f5d80032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 --2- 172.21.15.115:0/629775455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d980f5410 0x7f5d98106720 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:44.403 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 >> 172.21.15.115:0/629775455 conn(0x7f5d980f0b70 msgr2=0x7f5d9805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:44.404 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 shutdown_connections 2024-01-21T07:34:44.404 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:44.401+0000 7f5d9e19e700 1 -- 172.21.15.115:0/629775455 wait complete. 2024-01-21T07:34:44.404 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:45.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:44.400752+0000 mon.smithi107 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.115:0/629775455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:45.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:45.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:45.861938+0000 mgr.smithi107.adejht (mgr.14162) 173 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 -- 172.21.15.115:0/2390731438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 msgr2=0x7fe0900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:46.963 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 --2- 172.21.15.115:0/2390731438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe0900f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fe080009fb0 tx=0x7fe080009b30).stop 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 -- 172.21.15.115:0/2390731438 shutdown_connections 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 --2- 172.21.15.115:0/2390731438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe0900f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 -- 172.21.15.115:0/2390731438 >> 172.21.15.115:0/2390731438 conn(0x7fe0900f0b70 msgr2=0x7fe0900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 -- 172.21.15.115:0/2390731438 shutdown_connections 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.959+0000 7fe0960a1700 1 -- 172.21.15.115:0/2390731438 wait complete. 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe0960a1700 1 Processor -- start 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe0960a1700 1 -- start start 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe0960a1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe0960a1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090060220 con 0x7fe0900f5410 2024-01-21T07:34:46.964 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe08f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe08f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60016/0 (socket says 172.21.15.115:60016) 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.960+0000 7fe08f7fe700 1 -- 172.21.15.115:0/483117553 learned_addr learned my addr 172.21.15.115:0/483117553 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe08f7fe700 1 -- 172.21.15.115:0/483117553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe080005040 con 0x7fe0900f5410 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe08f7fe700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe080013510 tx=0x7fe08000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe080009320 con 0x7fe0900f5410 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe090060420 con 0x7fe0900f5410 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe080013f00 con 0x7fe0900f5410 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe080014d90 con 0x7fe0900f5410 2024-01-21T07:34:46.965 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.961+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe090109b00 con 0x7fe0900f5410 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0800263a0 con 0x7fe0900f5410 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe077fff700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090059100 con 0x7fe0900f5410 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe080041080 con 0x7fe0900f5410 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe08effd700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:46.966 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.962+0000 7fe08effd700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 0x7fe078032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe0840096a0 tx=0x7fe084006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:46.968 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:46.966+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe08006d2b0 con 0x7fe0900f5410 2024-01-21T07:34:47.177 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.175+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe090045ad0 con 0x7fe0900f5410 2024-01-21T07:34:47.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.175+0000 7fe077fff700 1 -- 172.21.15.115:0/483117553 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe080012510 con 0x7fe0900f5410 2024-01-21T07:34:47.178 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:47.178 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 msgr2=0x7fe078032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 0x7fe078032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe0840096a0 tx=0x7fe084006b40).stop 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 msgr2=0x7fe090109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe080013510 tx=0x7fe08000b040).stop 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 shutdown_connections 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 --2- 172.21.15.115:0/483117553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0900f5410 0x7fe090109450 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 >> 172.21.15.115:0/483117553 conn(0x7fe0900f0b70 msgr2=0x7fe0900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 shutdown_connections 2024-01-21T07:34:47.180 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:47.177+0000 7fe0960a1700 1 -- 172.21.15.115:0/483117553 wait complete. 2024-01-21T07:34:47.183 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:48.122 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:47.176937+0000 mon.smithi107 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.115:0/483117553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:48.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:48.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:49.023 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:47.862250+0000 mgr.smithi107.adejht (mgr.14162) 174 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:49.755 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- 172.21.15.115:0/2437862348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 msgr2=0x7f2a7c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/2437862348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c0f66c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f2a70009fb0 tx=0x7f2a70009b30).stop 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- 172.21.15.115:0/2437862348 shutdown_connections 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/2437862348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c0f66c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- 172.21.15.115:0/2437862348 >> 172.21.15.115:0/2437862348 conn(0x7f2a7c0f1aa0 msgr2=0x7f2a7c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- 172.21.15.115:0/2437862348 shutdown_connections 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- 172.21.15.115:0/2437862348 wait complete. 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 Processor -- start 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 -- start start 2024-01-21T07:34:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.752+0000 7f2a7bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7bfff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c108ba0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34758/0 (socket says 172.21.15.115:34758) 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7affd700 1 -- 172.21.15.115:0/1419195177 learned_addr learned my addr 172.21.15.115:0/1419195177 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7affd700 1 -- 172.21.15.115:0/1419195177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70005040 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7affd700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2a70013510 tx=0x7f2a7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a70009320 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c108da0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a70013f00 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a70014dc0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.753+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c109230 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.758 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.754+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a700095f0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.758 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.754+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c045ad0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.754+0000 7f2a637fe700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 0x7f2a64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.754+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a700410c0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.754+0000 7f2a7a7fc700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 0x7f2a64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.755+0000 7f2a7a7fc700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 0x7f2a64032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2a6c0096a0 tx=0x7f2a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:49.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.759+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a70014420 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.966+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a7c10a0e0 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.967+0000 7f2a637fe700 1 -- 172.21.15.115:0/1419195177 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2a70010230 con 0x7f2a7c0f62a0 2024-01-21T07:34:49.969 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:49.969 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:49.970 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 msgr2=0x7f2a64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 0x7f2a64032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2a6c0096a0 tx=0x7f2a6c006b40).stop 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 msgr2=0x7f2a7c1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2a70013510 tx=0x7f2a7000b040).stop 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 shutdown_connections 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2a6402ff60 0x7f2a64032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 --2- 172.21.15.115:0/1419195177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a7c0f62a0 0x7f2a7c1086b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 >> 172.21.15.115:0/1419195177 conn(0x7f2a7c0f1aa0 msgr2=0x7f2a7c0f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 shutdown_connections 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:49.968+0000 7f2a7bfff700 1 -- 172.21.15.115:0/1419195177 wait complete. 2024-01-21T07:34:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:51.067 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:49.640511+0000 mon.smithi107 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' 2024-01-21T07:34:51.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:34:49.641995+0000 mgr.smithi107.adejht (mgr.14162) 175 : cephadm [INF] Deploying daemon node-exporter.smithi107 on smithi107 2024-01-21T07:34:51.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:49.862513+0000 mgr.smithi107.adejht (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:51.068 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:49.968179+0000 mon.smithi107 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.115:0/1419195177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:51.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:51.348 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2203639054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580f5410 msgr2=0x7fbb580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:52.529 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2203639054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580f5410 0x7fbb580f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fbb48009fb0 tx=0x7fbb48009b30).stop 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2203639054 shutdown_connections 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2203639054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580f5410 0x7fbb580f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2203639054 >> 172.21.15.115:0/2203639054 conn(0x7fbb580f0b70 msgr2=0x7fbb580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2203639054 shutdown_connections 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.525+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2203639054 wait complete. 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5f6ec700 1 Processor -- start 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5f6ec700 1 -- start start 2024-01-21T07:34:52.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5f6ec700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5f6ec700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb580fd780 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5d488700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5d488700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34774/0 (socket says 172.21.15.115:34774) 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.526+0000 7fbb5d488700 1 -- 172.21.15.115:0/2383687464 learned_addr learned my addr 172.21.15.115:0/2383687464 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb5d488700 1 -- 172.21.15.115:0/2383687464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48005040 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb5d488700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fbb48009f80 tx=0x7fbb48009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb480136c0 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb580fbd10 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb480095f0 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb48014420 con 0x7fbb580fd360 2024-01-21T07:34:52.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.527+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb580fc200 con 0x7fbb580fd360 2024-01-21T07:34:52.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.528+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb48025060 con 0x7fbb580fd360 2024-01-21T07:34:52.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.528+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb58059100 con 0x7fbb580fd360 2024-01-21T07:34:52.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.528+0000 7fbb4dffb700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 0x7fbb44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:52.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.528+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb48040e40 con 0x7fbb580fd360 2024-01-21T07:34:52.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.528+0000 7fbb5cc87700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 0x7fbb44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:52.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.529+0000 7fbb5cc87700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 0x7fbb44032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbb540096a0 tx=0x7fbb54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:52.535 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.533+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb4801cb60 con 0x7fbb580fd360 2024-01-21T07:34:52.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.739+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb580fc5f0 con 0x7fbb580fd360 2024-01-21T07:34:52.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.739+0000 7fbb4dffb700 1 -- 172.21.15.115:0/2383687464 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb480422d0 con 0x7fbb580fd360 2024-01-21T07:34:52.741 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:52.742 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 msgr2=0x7fbb44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 0x7fbb44032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbb540096a0 tx=0x7fbb54006b40).stop 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 msgr2=0x7fbb580fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fbb48009f80 tx=0x7fbb48009b30).stop 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 shutdown_connections 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbb4402ff60 0x7fbb44032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 --2- 172.21.15.115:0/2383687464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb580fd360 0x7fbb580fb820 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 >> 172.21.15.115:0/2383687464 conn(0x7fbb580f0b70 msgr2=0x7fbb5805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:52.743 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 shutdown_connections 2024-01-21T07:34:52.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:52.741+0000 7fbb5f6ec700 1 -- 172.21.15.115:0/2383687464 wait complete. 2024-01-21T07:34:52.744 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:53.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:51.862796+0000 mgr.smithi107.adejht (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:53.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:52.740776+0000 mon.smithi107 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.115:0/2383687464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:54.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:54.143 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:53.863193+0000 mgr.smithi107.adejht (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- 172.21.15.115:0/3318841319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 msgr2=0x7f65a00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 --2- 172.21.15.115:0/3318841319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f9190 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f659c009fb0 tx=0x7f659c009b30).stop 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- 172.21.15.115:0/3318841319 shutdown_connections 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 --2- 172.21.15.115:0/3318841319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f9190 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- 172.21.15.115:0/3318841319 >> 172.21.15.115:0/3318841319 conn(0x7f65a00f0b70 msgr2=0x7f65a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- 172.21.15.115:0/3318841319 shutdown_connections 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- 172.21.15.115:0/3318841319 wait complete. 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 Processor -- start 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.306+0000 7f65a8778700 1 -- start start 2024-01-21T07:34:55.309 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a8778700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a8778700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00f5090 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a6514700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a6514700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34780/0 (socket says 172.21.15.115:34780) 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a6514700 1 -- 172.21.15.115:0/3438402274 learned_addr learned my addr 172.21.15.115:0/3438402274 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a6514700 1 -- 172.21.15.115:0/3438402274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c005040 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a6514700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f659c004dd0 tx=0x7f659c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659c018070 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a00f5230 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.307+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f659c013410 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a00f56c0 con 0x7f65a00601e0 2024-01-21T07:34:55.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659c01c3c0 con 0x7f65a00601e0 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f659c01d420 con 0x7f65a00601e0 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a0059100 con 0x7f65a00601e0 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f6596ffd700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 0x7f658c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.308+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f659c041420 con 0x7f65a00601e0 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.309+0000 7f65a5d13700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 0x7f658c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:55.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.309+0000 7f65a5d13700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 0x7f658c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f65900096a0 tx=0x7f6590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:55.314 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.313+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f659c01d820 con 0x7f65a00601e0 2024-01-21T07:34:55.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.517+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65a00fbf60 con 0x7f65a00601e0 2024-01-21T07:34:55.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.518+0000 7f6596ffd700 1 -- 172.21.15.115:0/3438402274 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f659c004490 con 0x7f65a00601e0 2024-01-21T07:34:55.519 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:55.519 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:55.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 msgr2=0x7f658c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 0x7f658c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f65900096a0 tx=0x7f6590006b40).stop 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 msgr2=0x7f65a00f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f659c004dd0 tx=0x7f659c009b30).stop 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 shutdown_connections 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f658c02ff60 0x7f658c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 --2- 172.21.15.115:0/3438402274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00601e0 0x7f65a00f6c10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 >> 172.21.15.115:0/3438402274 conn(0x7f65a00f0b70 msgr2=0x7f65a00f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 shutdown_connections 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:55.519+0000 7f65a8778700 1 -- 172.21.15.115:0/3438402274 wait complete. 2024-01-21T07:34:55.521 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:56.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:55.517931+0000 mon.smithi107 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.115:0/3438402274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:56.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:56.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:34:57.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:56 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:55.863655+0000 mgr.smithi107.adejht (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- 172.21.15.115:0/987797041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 msgr2=0x7f80fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 --2- 172.21.15.115:0/987797041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc0f66c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f80ec009fb0 tx=0x7f80ec009b30).stop 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- 172.21.15.115:0/987797041 shutdown_connections 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 --2- 172.21.15.115:0/987797041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc0f66c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- 172.21.15.115:0/987797041 >> 172.21.15.115:0/987797041 conn(0x7f80fc0f1aa0 msgr2=0x7f80fc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- 172.21.15.115:0/987797041 shutdown_connections 2024-01-21T07:34:58.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- 172.21.15.115:0/987797041 wait complete. 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 Processor -- start 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.037+0000 7f8102b19700 1 -- start start 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8102b19700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8102b19700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc069e20 con 0x7f80fc0f62a0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8101b17700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8101b17700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34792/0 (socket says 172.21.15.115:34792) 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8101b17700 1 -- 172.21.15.115:0/2397551481 learned_addr learned my addr 172.21.15.115:0/2397551481 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8101b17700 1 -- 172.21.15.115:0/2397551481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec005040 con 0x7f80fc0f62a0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.038+0000 7f8101b17700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f80ec013510 tx=0x7f80ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ec009320 con 0x7f80fc0f62a0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc069fc0 con 0x7f80fc0f62a0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc06a450 con 0x7f80fc0f62a0 2024-01-21T07:34:58.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80ec013f00 con 0x7f80fc0f62a0 2024-01-21T07:34:58.042 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ec014d90 con 0x7f80fc0f62a0 2024-01-21T07:34:58.042 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.039+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80ec014420 con 0x7f80fc0f62a0 2024-01-21T07:34:58.043 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.040+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e4004fa0 con 0x7f80fc0f62a0 2024-01-21T07:34:58.043 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.040+0000 7f80f27fc700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 0x7f80e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:34:58.043 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.040+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80ec041240 con 0x7f80fc0f62a0 2024-01-21T07:34:58.043 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.040+0000 7f8101316700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 0x7f80e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:34:58.043 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.040+0000 7f8101316700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 0x7f80e8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f80f80096a0 tx=0x7f80f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:34:58.045 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.044+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80ec00fb30 con 0x7f80fc0f62a0 2024-01-21T07:34:58.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.251+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80e4005670 con 0x7f80fc0f62a0 2024-01-21T07:34:58.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.252+0000 7f80f27fc700 1 -- 172.21.15.115:0/2397551481 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80ec00fb30 con 0x7f80fc0f62a0 2024-01-21T07:34:58.253 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:34:58.253 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 msgr2=0x7f80e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 0x7f80e8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f80f80096a0 tx=0x7f80f8006b40).stop 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 msgr2=0x7f80fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f80ec013510 tx=0x7f80ec00b040).stop 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 shutdown_connections 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f80e802ff60 0x7f80e8032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 --2- 172.21.15.115:0/2397551481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80fc0f62a0 0x7f80fc06b9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 >> 172.21.15.115:0/2397551481 conn(0x7f80fc0f1aa0 msgr2=0x7f80fc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:34:58.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 shutdown_connections 2024-01-21T07:34:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:34:58.254+0000 7f8102b19700 1 -- 172.21.15.115:0/2397551481 wait complete. 2024-01-21T07:34:58.256 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:34:58.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:57.863997+0000 mgr.smithi107.adejht (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:34:59.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:34:58.252469+0000 mon.smithi107 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.115:0/2397551481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:34:59.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:34:59.626 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:00.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:34:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:34:59.864404+0000 mgr.smithi107.adejht (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 -- 172.21.15.115:0/2718179857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 msgr2=0x7f86700f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 --2- 172.21.15.115:0/2718179857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f86700f4be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f8660009fb0 tx=0x7f8660009b30).stop 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 -- 172.21.15.115:0/2718179857 shutdown_connections 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 --2- 172.21.15.115:0/2718179857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f86700f4be0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 -- 172.21.15.115:0/2718179857 >> 172.21.15.115:0/2718179857 conn(0x7f86700f02e0 msgr2=0x7f86700f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.779+0000 7f86782b9700 1 -- 172.21.15.115:0/2718179857 shutdown_connections 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f86782b9700 1 -- 172.21.15.115:0/2718179857 wait complete. 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f86782b9700 1 Processor -- start 2024-01-21T07:35:00.784 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f86782b9700 1 -- start start 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f86782b9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f86782b9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8670106bf0 con 0x7f86700f47c0 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f8676055700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f8676055700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35984/0 (socket says 172.21.15.115:35984) 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.780+0000 7f8676055700 1 -- 172.21.15.115:0/1598917238 learned_addr learned my addr 172.21.15.115:0/1598917238 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f8676055700 1 -- 172.21.15.115:0/1598917238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660005040 con 0x7f86700f47c0 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f8676055700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8660013510 tx=0x7f866000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8660009320 con 0x7f86700f47c0 2024-01-21T07:35:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8670106df0 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8660013f00 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8660014d90 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.781+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8670107280 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.782+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86600263a0 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.782+0000 7f8666ffd700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 0x7f865c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.782+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8660041010 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.782+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8670045ad0 con 0x7f86700f47c0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.782+0000 7f8675854700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 0x7f865c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:00.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.783+0000 7f8675854700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 0x7f865c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f866c0096a0 tx=0x7f866c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:00.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.787+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8660012920 con 0x7f86700f47c0 2024-01-21T07:35:00.993 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.992+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8670108130 con 0x7f86700f47c0 2024-01-21T07:35:00.994 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.992+0000 7f8666ffd700 1 -- 172.21.15.115:0/1598917238 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f866002b020 con 0x7f86700f47c0 2024-01-21T07:35:00.994 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:00.994 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:00.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 msgr2=0x7f865c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:00.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 0x7f865c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f866c0096a0 tx=0x7f866c006b40).stop 2024-01-21T07:35:00.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 msgr2=0x7f8670106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8660013510 tx=0x7f866000b040).stop 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 shutdown_connections 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f865c02ff60 0x7f865c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 --2- 172.21.15.115:0/1598917238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86700f47c0 0x7f8670106700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 >> 172.21.15.115:0/1598917238 conn(0x7f86700f02e0 msgr2=0x7f86700f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 shutdown_connections 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:00.994+0000 7f86782b9700 1 -- 172.21.15.115:0/1598917238 wait complete. 2024-01-21T07:35:00.996 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:01.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:01 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:00.992921+0000 mon.smithi107 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.115:0/1598917238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:02.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:02.381 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:03.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:01.864807+0000 mgr.smithi107.adejht (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:03.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 -- 172.21.15.115:0/1012166965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 msgr2=0x7fa9300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:03.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 --2- 172.21.15.115:0/1012166965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa9300f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fa920009fb0 tx=0x7fa920009b30).stop 2024-01-21T07:35:03.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 -- 172.21.15.115:0/1012166965 shutdown_connections 2024-01-21T07:35:03.558 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 --2- 172.21.15.115:0/1012166965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa9300f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 -- 172.21.15.115:0/1012166965 >> 172.21.15.115:0/1012166965 conn(0x7fa9300f0b70 msgr2=0x7fa9300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 -- 172.21.15.115:0/1012166965 shutdown_connections 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.555+0000 7fa937c0d700 1 -- 172.21.15.115:0/1012166965 wait complete. 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa937c0d700 1 Processor -- start 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa937c0d700 1 -- start start 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa937c0d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa937c0d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930106c10 con 0x7fa9300f5410 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa9359a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa9359a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36002/0 (socket says 172.21.15.115:36002) 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.556+0000 7fa9359a9700 1 -- 172.21.15.115:0/2126018836 learned_addr learned my addr 172.21.15.115:0/2126018836 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa9359a9700 1 -- 172.21.15.115:0/2126018836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920005040 con 0x7fa9300f5410 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa9359a9700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa920013510 tx=0x7fa92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:03.559 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa920009320 con 0x7fa9300f5410 2024-01-21T07:35:03.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa930106e10 con 0x7fa9300f5410 2024-01-21T07:35:03.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa920013f00 con 0x7fa9300f5410 2024-01-21T07:35:03.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa920014d90 con 0x7fa9300f5410 2024-01-21T07:35:03.560 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.557+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9301072a0 con 0x7fa9300f5410 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.558+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9200263a0 con 0x7fa9300f5410 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.558+0000 7fa9267fc700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 0x7fa91c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.558+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa930045ad0 con 0x7fa9300f5410 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.558+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa920041300 con 0x7fa9300f5410 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.558+0000 7fa9351a8700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 0x7fa91c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:03.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.559+0000 7fa9351a8700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 0x7fa91c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fa92c0096a0 tx=0x7fa92c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:03.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.563+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa920012510 con 0x7fa9300f5410 2024-01-21T07:35:03.769 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.768+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9301080f0 con 0x7fa9300f5410 2024-01-21T07:35:03.770 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.769+0000 7fa9267fc700 1 -- 172.21.15.115:0/2126018836 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa92000f340 con 0x7fa9300f5410 2024-01-21T07:35:03.770 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:03.770 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 msgr2=0x7fa91c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 0x7fa91c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fa92c0096a0 tx=0x7fa92c006b40).stop 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 msgr2=0x7fa930106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa920013510 tx=0x7fa92000b040).stop 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 shutdown_connections 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa91c02ff60 0x7fa91c032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.770+0000 7fa937c0d700 1 --2- 172.21.15.115:0/2126018836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9300f5410 0x7fa930106720 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.771+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 >> 172.21.15.115:0/2126018836 conn(0x7fa9300f0b70 msgr2=0x7fa93005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.771+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 shutdown_connections 2024-01-21T07:35:03.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:03.771+0000 7fa937c0d700 1 -- 172.21.15.115:0/2126018836 wait complete. 2024-01-21T07:35:03.773 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:03.769462+0000 mon.smithi107 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.115:0/2126018836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:05.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:05.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:03.865282+0000 mgr.smithi107.adejht (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:05.865651+0000 mgr.smithi107.adejht (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:06.316 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 -- 172.21.15.115:0/2942482496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 msgr2=0x7f65100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 --2- 172.21.15.115:0/2942482496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65100f55c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f64fc009fb0 tx=0x7f64fc009b30).stop 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 -- 172.21.15.115:0/2942482496 shutdown_connections 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 --2- 172.21.15.115:0/2942482496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65100f55c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 -- 172.21.15.115:0/2942482496 >> 172.21.15.115:0/2942482496 conn(0x7f65100f09a0 msgr2=0x7f65100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 -- 172.21.15.115:0/2942482496 shutdown_connections 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.315+0000 7f650f59e700 1 -- 172.21.15.115:0/2942482496 wait complete. 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650f59e700 1 Processor -- start 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650f59e700 1 -- start start 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650f59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650f59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6510060240 con 0x7f65100f51a0 2024-01-21T07:35:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36010/0 (socket says 172.21.15.115:36010) 2024-01-21T07:35:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650e59c700 1 -- 172.21.15.115:0/2478385791 learned_addr learned my addr 172.21.15.115:0/2478385791 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.316+0000 7f650e59c700 1 -- 172.21.15.115:0/2478385791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc005040 con 0x7f65100f51a0 2024-01-21T07:35:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f650e59c700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f64fc009830 tx=0x7f64fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc009320 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6510060440 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6510109860 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64fc004490 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.317+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64fc01ee20 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64fc01e420 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec004fa0 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f6506ffd700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 0x7f64f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64fc040d30 con 0x7f65100f51a0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f650dd9b700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 0x7f64f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:06.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.318+0000 7f650dd9b700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 0x7f64f4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f65000096a0 tx=0x7f6500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:06.323 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.322+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64fc0095f0 con 0x7f65100f51a0 2024-01-21T07:35:06.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.529+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64ec005670 con 0x7f65100f51a0 2024-01-21T07:35:06.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.529+0000 7f6506ffd700 1 -- 172.21.15.115:0/2478385791 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f64fc02b2d0 con 0x7f65100f51a0 2024-01-21T07:35:06.531 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:06.531 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:06.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 msgr2=0x7f64f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:06.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 0x7f64f4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f65000096a0 tx=0x7f6500006b40).stop 2024-01-21T07:35:06.532 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 msgr2=0x7f65101091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f64fc009830 tx=0x7f64fc00b040).stop 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 shutdown_connections 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f64f402ff60 0x7f64f4032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 --2- 172.21.15.115:0/2478385791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65100f51a0 0x7f65101091b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 >> 172.21.15.115:0/2478385791 conn(0x7f65100f09a0 msgr2=0x7f65100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 shutdown_connections 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:06.531+0000 7f650f59e700 1 -- 172.21.15.115:0/2478385791 wait complete. 2024-01-21T07:35:06.533 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:07.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:06.529935+0000 mon.smithi107 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.115:0/2478385791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:07.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:07.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:07.865985+0000 mgr.smithi107.adejht (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 -- 172.21.15.115:0/2717942533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 msgr2=0x7f62940f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 --2- 172.21.15.115:0/2717942533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62940f6ea0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6288009fb0 tx=0x7f6288009b30).stop 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 -- 172.21.15.115:0/2717942533 shutdown_connections 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 --2- 172.21.15.115:0/2717942533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62940f6ea0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 -- 172.21.15.115:0/2717942533 >> 172.21.15.115:0/2717942533 conn(0x7f62940f02e0 msgr2=0x7f62940f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 -- 172.21.15.115:0/2717942533 shutdown_connections 2024-01-21T07:35:09.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.055+0000 7f629b188700 1 -- 172.21.15.115:0/2717942533 wait complete. 2024-01-21T07:35:09.060 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f629b188700 1 Processor -- start 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f629b188700 1 -- start start 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f629b188700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f629b188700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294106bc0 con 0x7f62940f6a80 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f6298f24700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f6298f24700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:36022/0 (socket says 172.21.15.115:36022) 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.056+0000 7f6298f24700 1 -- 172.21.15.115:0/569866097 learned_addr learned my addr 172.21.15.115:0/569866097 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f6298f24700 1 -- 172.21.15.115:0/569866097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6288005040 con 0x7f62940f6a80 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f6298f24700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6288013510 tx=0x7f628800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6288009320 con 0x7f62940f6a80 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6288013f00 con 0x7f62940f6a80 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6294106dc0 con 0x7f62940f6a80 2024-01-21T07:35:09.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6288014dc0 con 0x7f62940f6a80 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.057+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6294107250 con 0x7f62940f6a80 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.058+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62880095f0 con 0x7f62940f6a80 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.058+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6294045ad0 con 0x7f62940f6a80 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.058+0000 7f62917fa700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 0x7f6284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.058+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6288041100 con 0x7f62940f6a80 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.058+0000 7f6293fff700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 0x7f6284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:09.062 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.059+0000 7f6293fff700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 0x7f6284032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f62800096a0 tx=0x7f6280006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:09.064 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.063+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6288014420 con 0x7f62940f6a80 2024-01-21T07:35:09.268 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.266+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6294108100 con 0x7f62940f6a80 2024-01-21T07:35:09.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.267+0000 7f62917fa700 1 -- 172.21.15.115:0/569866097 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6288010230 con 0x7f62940f6a80 2024-01-21T07:35:09.269 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:09.269 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 msgr2=0x7f6284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 0x7f6284032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f62800096a0 tx=0x7f6280006b40).stop 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 msgr2=0x7f62941066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6288013510 tx=0x7f628800b040).stop 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 shutdown_connections 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f628402ff60 0x7f6284032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 --2- 172.21.15.115:0/569866097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62940f6a80 0x7f62941066d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.268+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 >> 172.21.15.115:0/569866097 conn(0x7f62940f02e0 msgr2=0x7f62940f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.269+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 shutdown_connections 2024-01-21T07:35:09.271 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:09.269+0000 7f629b188700 1 -- 172.21.15.115:0/569866097 wait complete. 2024-01-21T07:35:09.272 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:09.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:09.268392+0000 mon.smithi107 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.115:0/569866097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:10.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:10.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:10.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:09.866358+0000 mgr.smithi107.adejht (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:10.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:09.950161+0000 mon.smithi107 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:35:10.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:09.951267+0000 mon.smithi107 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 -- 172.21.15.115:0/3994909681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 msgr2=0x7f5fe40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 --2- 172.21.15.115:0/3994909681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f9150 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f5fd4009fb0 tx=0x7f5fd4009b30).stop 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 -- 172.21.15.115:0/3994909681 shutdown_connections 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 --2- 172.21.15.115:0/3994909681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f9150 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 -- 172.21.15.115:0/3994909681 >> 172.21.15.115:0/3994909681 conn(0x7f5fe40f0b70 msgr2=0x7f5fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:11.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 -- 172.21.15.115:0/3994909681 shutdown_connections 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.812+0000 7f5fec780700 1 -- 172.21.15.115:0/3994909681 wait complete. 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fec780700 1 Processor -- start 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fec780700 1 -- start start 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fec780700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fec780700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40f5050 con 0x7f5fe4060220 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fea51c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fea51c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39592/0 (socket says 172.21.15.115:39592) 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.813+0000 7f5fea51c700 1 -- 172.21.15.115:0/1000709445 learned_addr learned my addr 172.21.15.115:0/1000709445 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fea51c700 1 -- 172.21.15.115:0/1000709445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4005040 con 0x7f5fe4060220 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fea51c700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5fd4013510 tx=0x7f5fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fd4009320 con 0x7f5fe4060220 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe40f51f0 con 0x7f5fe4060220 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fd4013f00 con 0x7f5fe4060220 2024-01-21T07:35:11.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fd4014dc0 con 0x7f5fe4060220 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.814+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe40f5680 con 0x7f5fe4060220 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fd40095f0 con 0x7f5fe4060220 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe4059100 con 0x7f5fe4060220 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fdaffd700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 0x7f5fd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fd4041100 con 0x7f5fe4060220 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fe9d1b700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 0x7f5fd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:11.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.815+0000 7f5fe9d1b700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 0x7f5fd0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5fe00096a0 tx=0x7f5fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:11.822 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:11.820+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fd4014420 con 0x7f5fe4060220 2024-01-21T07:35:12.025 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.023+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fe40fbf50 con 0x7f5fe4060220 2024-01-21T07:35:12.026 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.024+0000 7f5fdaffd700 1 -- 172.21.15.115:0/1000709445 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5fd4010230 con 0x7f5fe4060220 2024-01-21T07:35:12.026 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:12.026 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 msgr2=0x7f5fd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 0x7f5fd0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5fe00096a0 tx=0x7f5fe0006b40).stop 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 msgr2=0x7f5fe40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5fd4013510 tx=0x7f5fd400b040).stop 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 shutdown_connections 2024-01-21T07:35:12.028 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5fd002ff60 0x7f5fd0032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:12.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 --2- 172.21.15.115:0/1000709445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe4060220 0x7f5fe40f6bd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:12.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 >> 172.21.15.115:0/1000709445 conn(0x7f5fe40f0b70 msgr2=0x7f5fe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:12.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 shutdown_connections 2024-01-21T07:35:12.029 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:12.025+0000 7f5fec780700 1 -- 172.21.15.115:0/1000709445 wait complete. 2024-01-21T07:35:12.030 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:13.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:11.866727+0000 mgr.smithi107.adejht (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:13.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:12.025286+0000 mon.smithi107 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.115:0/1000709445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:13.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:13.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:14.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:13.867067+0000 mgr.smithi107.adejht (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/538412191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 msgr2=0x7f8b280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/538412191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b280f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f8b10009fb0 tx=0x7f8b10009b30).stop 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/538412191 shutdown_connections 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/538412191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b280f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/538412191 >> 172.21.15.115:0/538412191 conn(0x7f8b280f0b70 msgr2=0x7f8b280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/538412191 shutdown_connections 2024-01-21T07:35:14.605 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.602+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/538412191 wait complete. 2024-01-21T07:35:14.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b2d2d5700 1 Processor -- start 2024-01-21T07:35:14.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b2d2d5700 1 -- start start 2024-01-21T07:35:14.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b2d2d5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:14.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b2d2d5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b28060240 con 0x7f8b280f5410 2024-01-21T07:35:14.607 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b26ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b26ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39610/0 (socket says 172.21.15.115:39610) 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.603+0000 7f8b26ffd700 1 -- 172.21.15.115:0/1166251402 learned_addr learned my addr 172.21.15.115:0/1166251402 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b26ffd700 1 -- 172.21.15.115:0/1166251402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10005040 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b26ffd700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8b10013510 tx=0x7f8b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b10009320 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b10013f00 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b28060440 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b10014d90 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.604+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b28109ae0 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.605+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b100263a0 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.605+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b28045ad0 con 0x7f8b280f5410 2024-01-21T07:35:14.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.605+0000 7f8b1f7fe700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 0x7f8b14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:14.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.605+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b10041390 con 0x7f8b280f5410 2024-01-21T07:35:14.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.605+0000 7f8b267fc700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 0x7f8b14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:14.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.606+0000 7f8b267fc700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 0x7f8b14032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8b180096a0 tx=0x7f8b18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:14.612 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.610+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b1001cb10 con 0x7f8b280f5410 2024-01-21T07:35:14.817 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.815+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b280fae60 con 0x7f8b280f5410 2024-01-21T07:35:14.818 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.816+0000 7f8b1f7fe700 1 -- 172.21.15.115:0/1166251402 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b1001cb10 con 0x7f8b280f5410 2024-01-21T07:35:14.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:14.818 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:14.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 msgr2=0x7f8b14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:14.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 0x7f8b14032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8b180096a0 tx=0x7f8b18006b40).stop 2024-01-21T07:35:14.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 msgr2=0x7f8b28109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:14.819 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8b10013510 tx=0x7f8b1000b040).stop 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 shutdown_connections 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f8b1402ff60 0x7f8b14032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 --2- 172.21.15.115:0/1166251402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b280f5410 0x7f8b28109430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 >> 172.21.15.115:0/1166251402 conn(0x7f8b280f0b70 msgr2=0x7f8b280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 shutdown_connections 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:14.817+0000 7f8b2d2d5700 1 -- 172.21.15.115:0/1166251402 wait complete. 2024-01-21T07:35:14.820 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:15.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:14.817251+0000 mon.smithi107 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.115:0/1166251402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:16.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:16.205 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:15.867460+0000 mgr.smithi107.adejht (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 -- 172.21.15.115:0/1581131634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 msgr2=0x7f423c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 --2- 172.21.15.115:0/1581131634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f9150 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f422c009fb0 tx=0x7f422c009b30).stop 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 -- 172.21.15.115:0/1581131634 shutdown_connections 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 --2- 172.21.15.115:0/1581131634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f9150 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 -- 172.21.15.115:0/1581131634 >> 172.21.15.115:0/1581131634 conn(0x7f423c0f0b70 msgr2=0x7f423c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 -- 172.21.15.115:0/1581131634 shutdown_connections 2024-01-21T07:35:17.412 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.408+0000 7f4240cfd700 1 -- 172.21.15.115:0/1581131634 wait complete. 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f4240cfd700 1 Processor -- start 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f4240cfd700 1 -- start start 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f4240cfd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f4240cfd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0f5050 con 0x7f423c060220 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f423a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f423a59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:39632/0 (socket says 172.21.15.115:39632) 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f423a59c700 1 -- 172.21.15.115:0/3022045342 learned_addr learned my addr 172.21.15.115:0/3022045342 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.409+0000 7f423a59c700 1 -- 172.21.15.115:0/3022045342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c005040 con 0x7f423c060220 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f423a59c700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f422c013510 tx=0x7f422c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f422c009320 con 0x7f423c060220 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c0f51f0 con 0x7f423c060220 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f422c013f00 con 0x7f423c060220 2024-01-21T07:35:17.413 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f422c014dc0 con 0x7f423c060220 2024-01-21T07:35:17.414 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.410+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c0f5680 con 0x7f423c060220 2024-01-21T07:35:17.414 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.411+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f422c0095f0 con 0x7f423c060220 2024-01-21T07:35:17.414 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.411+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423c059100 con 0x7f423c060220 2024-01-21T07:35:17.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.411+0000 7f422affd700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 0x7f4224032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:17.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.411+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f422c041020 con 0x7f423c060220 2024-01-21T07:35:17.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.411+0000 7f4239d9b700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 0x7f4224032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:17.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.412+0000 7f4239d9b700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 0x7f4224032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f42300096a0 tx=0x7f4230006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:17.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.416+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f422c014420 con 0x7f423c060220 2024-01-21T07:35:17.622 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.620+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f423c045ad0 con 0x7f423c060220 2024-01-21T07:35:17.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.620+0000 7f422affd700 1 -- 172.21.15.115:0/3022045342 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f422c01c920 con 0x7f423c060220 2024-01-21T07:35:17.623 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:17.623 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:17.624 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 msgr2=0x7f4224032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:17.624 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 0x7f4224032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f42300096a0 tx=0x7f4230006b40).stop 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 msgr2=0x7f423c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f422c013510 tx=0x7f422c00b040).stop 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 shutdown_connections 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f422402ff60 0x7f4224032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 --2- 172.21.15.115:0/3022045342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f423c060220 0x7f423c0f6bd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 >> 172.21.15.115:0/3022045342 conn(0x7f423c0f0b70 msgr2=0x7f423c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 shutdown_connections 2024-01-21T07:35:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:17.622+0000 7f4240cfd700 1 -- 172.21.15.115:0/3022045342 wait complete. 2024-01-21T07:35:17.626 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:17.621871+0000 mon.smithi107 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.115:0/3022045342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:19.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:19.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:19.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:17.867705+0000 mgr.smithi107.adejht (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 -- 172.21.15.115:0/51616276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 msgr2=0x7f515c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 --2- 172.21.15.115:0/51616276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c0f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f514c009fb0 tx=0x7f514c009b30).stop 2024-01-21T07:35:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 -- 172.21.15.115:0/51616276 shutdown_connections 2024-01-21T07:35:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 --2- 172.21.15.115:0/51616276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c0f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:20.178 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 -- 172.21.15.115:0/51616276 >> 172.21.15.115:0/51616276 conn(0x7f515c0f1c70 msgr2=0x7f515c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 -- 172.21.15.115:0/51616276 shutdown_connections 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.175+0000 7f51638ba700 1 -- 172.21.15.115:0/51616276 wait complete. 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f51638ba700 1 Processor -- start 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f51638ba700 1 -- start start 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f51638ba700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f51638ba700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c106c20 con 0x7f515c0f6510 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f5161656700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f5161656700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35476/0 (socket says 172.21.15.115:35476) 2024-01-21T07:35:20.179 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.176+0000 7f5161656700 1 -- 172.21.15.115:0/2221826471 learned_addr learned my addr 172.21.15.115:0/2221826471 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f5161656700 1 -- 172.21.15.115:0/2221826471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c005040 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f5161656700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f514c014000 tx=0x7f514c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f514c0095f0 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c106e20 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f514c007e70 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f514c014ea0 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.177+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c1072b0 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.178+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f514c025070 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.178+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c045ad0 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.178+0000 7f5151ffb700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 0x7f5148032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.178+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f514c040990 con 0x7f515c0f6510 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.178+0000 7f5160e55700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 0x7f5148032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:20.181 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.179+0000 7f5160e55700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 0x7f5148032470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f51580096a0 tx=0x7f5158006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:20.185 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.183+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f514c0079c0 con 0x7f515c0f6510 2024-01-21T07:35:20.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:19.868041+0000 mgr.smithi107.adejht (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.389+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f515c0fbf60 con 0x7f515c0f6510 2024-01-21T07:35:20.391 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.389+0000 7f5151ffb700 1 -- 172.21.15.115:0/2221826471 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f514c00f300 con 0x7f515c0f6510 2024-01-21T07:35:20.391 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:20.392 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:20.393 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 msgr2=0x7f5148032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 0x7f5148032470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f51580096a0 tx=0x7f5158006b40).stop 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 msgr2=0x7f515c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f514c014000 tx=0x7f514c00b040).stop 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 shutdown_connections 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f514802ffb0 0x7f5148032470 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 --2- 172.21.15.115:0/2221826471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f515c0f6510 0x7f515c106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 >> 172.21.15.115:0/2221826471 conn(0x7f515c0f1c70 msgr2=0x7f515c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 shutdown_connections 2024-01-21T07:35:20.394 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:20.391+0000 7f51638ba700 1 -- 172.21.15.115:0/2221826471 wait complete. 2024-01-21T07:35:20.395 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:21.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:20.390790+0000 mon.smithi107 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.115:0/2221826471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:21.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:21.776 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 -- 172.21.15.115:0/2355687392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 msgr2=0x7f66100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 --2- 172.21.15.115:0/2355687392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66100f90f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f6600009fb0 tx=0x7f6600009b30).stop 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 -- 172.21.15.115:0/2355687392 shutdown_connections 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 --2- 172.21.15.115:0/2355687392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66100f90f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 -- 172.21.15.115:0/2355687392 >> 172.21.15.115:0/2355687392 conn(0x7f66100f0b70 msgr2=0x7f66100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 -- 172.21.15.115:0/2355687392 shutdown_connections 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.924+0000 7f661756d700 1 -- 172.21.15.115:0/2355687392 wait complete. 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f661756d700 1 Processor -- start 2024-01-21T07:35:22.928 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f661756d700 1 -- start start 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f661756d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f661756d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6610106bd0 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f6615309700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f6615309700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35496/0 (socket says 172.21.15.115:35496) 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f6615309700 1 -- 172.21.15.115:0/2816912068 learned_addr learned my addr 172.21.15.115:0/2816912068 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f6615309700 1 -- 172.21.15.115:0/2816912068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600005040 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.925+0000 7f6615309700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6600013510 tx=0x7f660000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.926+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6600009320 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.926+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6610106dd0 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.926+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6600013f00 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.926+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6600014dc0 con 0x7f6610060280 2024-01-21T07:35:22.929 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.926+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6610107260 con 0x7f6610060280 2024-01-21T07:35:22.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f660001d3d0 con 0x7f6610060280 2024-01-21T07:35:22.930 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6610045ad0 con 0x7f6610060280 2024-01-21T07:35:22.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f6605ffb700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 0x7f65fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:22.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66000413f0 con 0x7f6610060280 2024-01-21T07:35:22.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f6614b08700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 0x7f65fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:22.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.927+0000 7f6614b08700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 0x7f65fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f660c0096a0 tx=0x7f660c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:22.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:22.932+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66000126f0 con 0x7f6610060280 2024-01-21T07:35:23.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.140+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66101080b0 con 0x7f6610060280 2024-01-21T07:35:23.143 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.140+0000 7f6605ffb700 1 -- 172.21.15.115:0/2816912068 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6600012330 con 0x7f6610060280 2024-01-21T07:35:23.143 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:23.143 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:23.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 msgr2=0x7f65fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:23.144 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 0x7f65fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f660c0096a0 tx=0x7f660c006b40).stop 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 msgr2=0x7f66101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6600013510 tx=0x7f660000b040).stop 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 shutdown_connections 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f65fc02ff60 0x7f65fc032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 --2- 172.21.15.115:0/2816912068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6610060280 0x7f66101066e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 >> 172.21.15.115:0/2816912068 conn(0x7f66100f0b70 msgr2=0x7f66100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 shutdown_connections 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:23.142+0000 7f661756d700 1 -- 172.21.15.115:0/2816912068 wait complete. 2024-01-21T07:35:23.145 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:21.868472+0000 mgr.smithi107.adejht (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:23.141852+0000 mon.smithi107 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.115:0/2816912068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:24.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:24.566 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:23.868807+0000 mgr.smithi107.adejht (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.715+0000 7fdbd3078700 1 -- 172.21.15.115:0/1638030436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 msgr2=0x7fdbcc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.715+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1638030436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc0f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fdbc0009fb0 tx=0x7fdbc0009b30).stop 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 -- 172.21.15.115:0/1638030436 shutdown_connections 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1638030436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc0f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 -- 172.21.15.115:0/1638030436 >> 172.21.15.115:0/1638030436 conn(0x7fdbcc0f1c70 msgr2=0x7fdbcc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 -- 172.21.15.115:0/1638030436 shutdown_connections 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 -- 172.21.15.115:0/1638030436 wait complete. 2024-01-21T07:35:25.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 Processor -- start 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.716+0000 7fdbd3078700 1 -- start start 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd3078700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd3078700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc1049e0 con 0x7fdbcc0f6510 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd0e14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd0e14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35502/0 (socket says 172.21.15.115:35502) 2024-01-21T07:35:25.720 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd0e14700 1 -- 172.21.15.115:0/1772573423 learned_addr learned my addr 172.21.15.115:0/1772573423 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:25.721 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd0e14700 1 -- 172.21.15.115:0/1772573423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0005040 con 0x7fdbcc0f6510 2024-01-21T07:35:25.721 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.717+0000 7fdbd0e14700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fdbc0013510 tx=0x7fdbc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc0009320 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc104be0 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbc0013f00 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc0014d90 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc105070 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbc00095f0 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.718+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc045ad0 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.719+0000 7fdbc97fa700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 0x7fdbbc036870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.719+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbc0040f90 con 0x7fdbcc0f6510 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.719+0000 7fdbcbfff700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 0x7fdbbc036870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:25.722 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.719+0000 7fdbcbfff700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 0x7fdbbc036870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdbb80096a0 tx=0x7fdbb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:25.725 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.723+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbc0010230 con 0x7fdbcc0f6510 2024-01-21T07:35:25.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.928+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbcc106690 con 0x7fdbcc0f6510 2024-01-21T07:35:25.931 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.929+0000 7fdbc97fa700 1 -- 172.21.15.115:0/1772573423 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdbc0010230 con 0x7fdbcc0f6510 2024-01-21T07:35:25.931 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:25.932 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 msgr2=0x7fdbbc036870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 0x7fdbbc036870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdbb80096a0 tx=0x7fdbb8006b40).stop 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 msgr2=0x7fdbcc1044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fdbc0013510 tx=0x7fdbc000b040).stop 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 shutdown_connections 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdbbc0343b0 0x7fdbbc036870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:25.933 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 --2- 172.21.15.115:0/1772573423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbcc0f6510 0x7fdbcc1044f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:25.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 >> 172.21.15.115:0/1772573423 conn(0x7fdbcc0f1c70 msgr2=0x7fdbcc0f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:25.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 shutdown_connections 2024-01-21T07:35:25.934 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:25.931+0000 7fdbd3078700 1 -- 172.21.15.115:0/1772573423 wait complete. 2024-01-21T07:35:25.934 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:25.869105+0000 mgr.smithi107.adejht (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:25.930649+0000 mon.smithi107 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.115:0/1772573423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:27.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:27.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:27.869493+0000 mgr.smithi107.adejht (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:28.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 -- 172.21.15.115:0/2291338365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 msgr2=0x7fe1740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:28.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 --2- 172.21.15.115:0/2291338365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe1740f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fe170009fb0 tx=0x7fe170009b30).stop 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 -- 172.21.15.115:0/2291338365 shutdown_connections 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 --2- 172.21.15.115:0/2291338365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe1740f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 -- 172.21.15.115:0/2291338365 >> 172.21.15.115:0/2291338365 conn(0x7fe1740f0b70 msgr2=0x7fe1740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 -- 172.21.15.115:0/2291338365 shutdown_connections 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.500+0000 7fe17b299700 1 -- 172.21.15.115:0/2291338365 wait complete. 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe17b299700 1 Processor -- start 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe17b299700 1 -- start start 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe17b299700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe17b299700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174106c10 con 0x7fe1740f5410 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe179035700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe179035700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35516/0 (socket says 172.21.15.115:35516) 2024-01-21T07:35:28.507 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe179035700 1 -- 172.21.15.115:0/2693864471 learned_addr learned my addr 172.21.15.115:0/2693864471 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe179035700 1 -- 172.21.15.115:0/2693864471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe170005040 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.501+0000 7fe179035700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fe170013510 tx=0x7fe17000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.502+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe170009320 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.502+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe174106e10 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.502+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe170013f00 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.502+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe170014d90 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.502+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1741072a0 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.503+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1700263a0 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.503+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174045ad0 con 0x7fe1740f5410 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.503+0000 7fe169ffb700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 0x7fe160032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:28.508 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.503+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1700412a0 con 0x7fe1740f5410 2024-01-21T07:35:28.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.503+0000 7fe178834700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 0x7fe160032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:28.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.504+0000 7fe178834700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 0x7fe160032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe1640096a0 tx=0x7fe164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:28.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.508+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe17001cb10 con 0x7fe1740f5410 2024-01-21T07:35:28.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.714+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe174108150 con 0x7fe1740f5410 2024-01-21T07:35:28.716 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.715+0000 7fe169ffb700 1 -- 172.21.15.115:0/2693864471 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe17001cb10 con 0x7fe1740f5410 2024-01-21T07:35:28.717 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:28.717 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 msgr2=0x7fe160032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 0x7fe160032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe1640096a0 tx=0x7fe164006b40).stop 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 msgr2=0x7fe174106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fe170013510 tx=0x7fe17000b040).stop 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 shutdown_connections 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe16002ffb0 0x7fe160032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 --2- 172.21.15.115:0/2693864471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1740f5410 0x7fe174106720 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 >> 172.21.15.115:0/2693864471 conn(0x7fe1740f0b70 msgr2=0x7fe17405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 shutdown_connections 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:28.716+0000 7fe17b299700 1 -- 172.21.15.115:0/2693864471 wait complete. 2024-01-21T07:35:28.719 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:29.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:28.716094+0000 mon.smithi107 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.115:0/2693864471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:30.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:30.135 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:29.869777+0000 mgr.smithi107.adejht (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:31.291 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- 172.21.15.115:0/160555195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 msgr2=0x7f25bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:31.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 --2- 172.21.15.115:0/160555195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc0f55c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f25b8009fb0 tx=0x7f25b8009b30).stop 2024-01-21T07:35:31.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- 172.21.15.115:0/160555195 shutdown_connections 2024-01-21T07:35:31.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 --2- 172.21.15.115:0/160555195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc0f55c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:31.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- 172.21.15.115:0/160555195 >> 172.21.15.115:0/160555195 conn(0x7f25bc0f09a0 msgr2=0x7f25bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:31.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- 172.21.15.115:0/160555195 shutdown_connections 2024-01-21T07:35:31.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- 172.21.15.115:0/160555195 wait complete. 2024-01-21T07:35:31.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 Processor -- start 2024-01-21T07:35:31.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- start start 2024-01-21T07:35:31.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:31.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c23a1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc1069d0 con 0x7f25bc0f51a0 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.285+0000 7f25c139f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c139f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:52882/0 (socket says 172.21.15.115:52882) 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c139f700 1 -- 172.21.15.115:0/2822715762 learned_addr learned my addr 172.21.15.115:0/2822715762 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c139f700 1 -- 172.21.15.115:0/2822715762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b8005040 con 0x7f25bc0f51a0 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c139f700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f25b8013510 tx=0x7f25b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b8009320 con 0x7f25bc0f51a0 2024-01-21T07:35:31.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc106bd0 con 0x7f25bc0f51a0 2024-01-21T07:35:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc107060 con 0x7f25bc0f51a0 2024-01-21T07:35:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25b8013f00 con 0x7f25bc0f51a0 2024-01-21T07:35:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.286+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25b8014d90 con 0x7f25bc0f51a0 2024-01-21T07:35:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.287+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25b8014420 con 0x7f25bc0f51a0 2024-01-21T07:35:31.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.287+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc045ad0 con 0x7f25bc0f51a0 2024-01-21T07:35:31.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.287+0000 7f25b1ffb700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 0x7f25a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:31.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.287+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25b8041310 con 0x7f25bc0f51a0 2024-01-21T07:35:31.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.288+0000 7f25c0b9e700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 0x7f25a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:31.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.288+0000 7f25c0b9e700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 0x7f25a8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f25ac0096a0 tx=0x7f25ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:31.297 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.292+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25b8012b20 con 0x7f25bc0f51a0 2024-01-21T07:35:31.501 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.499+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25bc107f10 con 0x7f25bc0f51a0 2024-01-21T07:35:31.502 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.500+0000 7f25b1ffb700 1 -- 172.21.15.115:0/2822715762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f25b8012b20 con 0x7f25bc0f51a0 2024-01-21T07:35:31.502 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:31.503 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:31.504 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 msgr2=0x7f25a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:31.504 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 0x7f25a8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f25ac0096a0 tx=0x7f25ac006b40).stop 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 msgr2=0x7f25bc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f25b8013510 tx=0x7f25b800b040).stop 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 shutdown_connections 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25a802ff60 0x7f25a8032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 --2- 172.21.15.115:0/2822715762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25bc0f51a0 0x7f25bc1064e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 >> 172.21.15.115:0/2822715762 conn(0x7f25bc0f09a0 msgr2=0x7f25bc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:31.505 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.501+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 shutdown_connections 2024-01-21T07:35:31.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:31.502+0000 7f25c23a1700 1 -- 172.21.15.115:0/2822715762 wait complete. 2024-01-21T07:35:31.506 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:31.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:31 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:31.501102+0000 mon.smithi107 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.115:0/2822715762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:32.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:32.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:33.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:31.869997+0000 mgr.smithi107.adejht (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:34.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 -- 172.21.15.115:0/1700797645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 msgr2=0x7efda00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:34.032 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 --2- 172.21.15.115:0/1700797645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda00f90f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7efd9c009fb0 tx=0x7efd9c009b30).stop 2024-01-21T07:35:34.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 -- 172.21.15.115:0/1700797645 shutdown_connections 2024-01-21T07:35:34.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 --2- 172.21.15.115:0/1700797645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda00f90f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:34.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 -- 172.21.15.115:0/1700797645 >> 172.21.15.115:0/1700797645 conn(0x7efda00f0b70 msgr2=0x7efda00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:34.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 -- 172.21.15.115:0/1700797645 shutdown_connections 2024-01-21T07:35:34.033 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.030+0000 7efda7c8e700 1 -- 172.21.15.115:0/1700797645 wait complete. 2024-01-21T07:35:34.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda7c8e700 1 Processor -- start 2024-01-21T07:35:34.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda7c8e700 1 -- start start 2024-01-21T07:35:34.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda7c8e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:34.034 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda7c8e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda0106bd0 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda5a2a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda5a2a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:52896/0 (socket says 172.21.15.115:52896) 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.031+0000 7efda5a2a700 1 -- 172.21.15.115:0/4257870649 learned_addr learned my addr 172.21.15.115:0/4257870649 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efda5a2a700 1 -- 172.21.15.115:0/4257870649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c005040 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efda5a2a700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7efd9c013510 tx=0x7efd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd9c009320 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda0106dd0 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd9c013f00 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd9c014dc0 con 0x7efda0060280 2024-01-21T07:35:34.035 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.032+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda0107260 con 0x7efda0060280 2024-01-21T07:35:34.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd9c01d3d0 con 0x7efda0060280 2024-01-21T07:35:34.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda0045ad0 con 0x7efda0060280 2024-01-21T07:35:34.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efd967fc700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 0x7efd8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:34.036 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd9c040fc0 con 0x7efda0060280 2024-01-21T07:35:34.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efda5229700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 0x7efd8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:34.037 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.033+0000 7efda5229700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 0x7efd8c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7efd900096a0 tx=0x7efd90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:34.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.038+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd9c01c790 con 0x7efda0060280 2024-01-21T07:35:34.244 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.242+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efda0108110 con 0x7efda0060280 2024-01-21T07:35:34.245 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.243+0000 7efd967fc700 1 -- 172.21.15.115:0/4257870649 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd9c01c790 con 0x7efda0060280 2024-01-21T07:35:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:34.245 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:34.247 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 msgr2=0x7efd8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:34.247 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 0x7efd8c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7efd900096a0 tx=0x7efd90006b40).stop 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 msgr2=0x7efda01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7efd9c013510 tx=0x7efd9c00b040).stop 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 shutdown_connections 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7efd8c02ff60 0x7efd8c032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 --2- 172.21.15.115:0/4257870649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efda0060280 0x7efda01066e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 >> 172.21.15.115:0/4257870649 conn(0x7efda00f0b70 msgr2=0x7efda00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 shutdown_connections 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:34.245+0000 7efda7c8e700 1 -- 172.21.15.115:0/4257870649 wait complete. 2024-01-21T07:35:34.248 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:33.870239+0000 mgr.smithi107.adejht (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:35.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:34.244674+0000 mon.smithi107 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.115:0/4257870649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:35.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:35.645 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:35.870523+0000 mgr.smithi107.adejht (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 -- 172.21.15.115:0/4118084131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 msgr2=0x7fe3e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/4118084131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00f55c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe3d0009fb0 tx=0x7fe3d0009b30).stop 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 -- 172.21.15.115:0/4118084131 shutdown_connections 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/4118084131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00f55c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 -- 172.21.15.115:0/4118084131 >> 172.21.15.115:0/4118084131 conn(0x7fe3e00f09a0 msgr2=0x7fe3e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 -- 172.21.15.115:0/4118084131 shutdown_connections 2024-01-21T07:35:36.777 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.773+0000 7fe3e6a00700 1 -- 172.21.15.115:0/4118084131 wait complete. 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e6a00700 1 Processor -- start 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e6a00700 1 -- start start 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e6a00700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e6a00700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00fb5a0 con 0x7fe3e00f51a0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e59fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e59fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:52916/0 (socket says 172.21.15.115:52916) 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e59fe700 1 -- 172.21.15.115:0/2784059192 learned_addr learned my addr 172.21.15.115:0/2784059192 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e59fe700 1 -- 172.21.15.115:0/2784059192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0005040 con 0x7fe3e00f51a0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.774+0000 7fe3e59fe700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe3d0013510 tx=0x7fe3d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d0009320 con 0x7fe3e00f51a0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e00fb740 con 0x7fe3e00f51a0 2024-01-21T07:35:36.778 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3d0013f00 con 0x7fe3e00f51a0 2024-01-21T07:35:36.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3d0014dc0 con 0x7fe3e00f51a0 2024-01-21T07:35:36.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e00fbbd0 con 0x7fe3e00f51a0 2024-01-21T07:35:36.779 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.775+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3d001d3d0 con 0x7fe3e00f51a0 2024-01-21T07:35:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.776+0000 7fe3d67fc700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 0x7fe3cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.776+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3d0041360 con 0x7fe3e00f51a0 2024-01-21T07:35:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.776+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e0059100 con 0x7fe3e00f51a0 2024-01-21T07:35:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.776+0000 7fe3e51fd700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 0x7fe3cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:36.780 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.776+0000 7fe3e51fd700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 0x7fe3cc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:36.782 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.780+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3d0012510 con 0x7fe3e00f51a0 2024-01-21T07:35:36.988 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.985+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3e00fabe0 con 0x7fe3e00f51a0 2024-01-21T07:35:36.988 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.986+0000 7fe3d67fc700 1 -- 172.21.15.115:0/2784059192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe3d002b030 con 0x7fe3e00f51a0 2024-01-21T07:35:36.988 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:36.989 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:36.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 msgr2=0x7fe3cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:36.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 0x7fe3cc032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe3dc0096a0 tx=0x7fe3dc006b40).stop 2024-01-21T07:35:36.990 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 msgr2=0x7fe3e00fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe3d0013510 tx=0x7fe3d000b040).stop 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 shutdown_connections 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe3cc02ff60 0x7fe3cc032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 --2- 172.21.15.115:0/2784059192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3e00f51a0 0x7fe3e00fd120 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 >> 172.21.15.115:0/2784059192 conn(0x7fe3e00f09a0 msgr2=0x7fe3e005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 shutdown_connections 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:36.988+0000 7fe3e6a00700 1 -- 172.21.15.115:0/2784059192 wait complete. 2024-01-21T07:35:36.991 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:36.987746+0000 mon.smithi107 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.115:0/2784059192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:38.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:38.372 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:37.870854+0000 mgr.smithi107.adejht (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:39.540 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 -- 172.21.15.115:0/638530870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 msgr2=0x7fe9fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:39.540 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 --2- 172.21.15.115:0/638530870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc0f6930 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fe9ec009fb0 tx=0x7fe9ec009b30).stop 2024-01-21T07:35:39.540 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 -- 172.21.15.115:0/638530870 shutdown_connections 2024-01-21T07:35:39.540 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 --2- 172.21.15.115:0/638530870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc0f6930 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 -- 172.21.15.115:0/638530870 >> 172.21.15.115:0/638530870 conn(0x7fe9fc0f1c70 msgr2=0x7fe9fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 -- 172.21.15.115:0/638530870 shutdown_connections 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.537+0000 7fea03867700 1 -- 172.21.15.115:0/638530870 wait complete. 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea03867700 1 Processor -- start 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea03867700 1 -- start start 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea03867700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea03867700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc069e20 con 0x7fe9fc0f6510 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea01603700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea01603700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:46542/0 (socket says 172.21.15.115:46542) 2024-01-21T07:35:39.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.538+0000 7fea01603700 1 -- 172.21.15.115:0/2580784982 learned_addr learned my addr 172.21.15.115:0/2580784982 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fea01603700 1 -- 172.21.15.115:0/2580784982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec005040 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fea01603700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe9ec013510 tx=0x7fe9ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9ec009320 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc069fc0 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9ec013f00 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9ec014d90 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.539+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc06a450 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.540+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9ec014420 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.540+0000 7fe9f1ffb700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 0x7fe9e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.540+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc059100 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.540+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9ec0410c0 con 0x7fe9fc0f6510 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.540+0000 7fea00e02700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 0x7fe9e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:39.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.541+0000 7fea00e02700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 0x7fe9e8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe9f80096a0 tx=0x7fe9f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:39.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.545+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9ec06c2b0 con 0x7fe9fc0f6510 2024-01-21T07:35:39.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.751+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9fc0fbf60 con 0x7fe9fc0f6510 2024-01-21T07:35:39.753 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.751+0000 7fe9f1ffb700 1 -- 172.21.15.115:0/2580784982 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9ec02c090 con 0x7fe9fc0f6510 2024-01-21T07:35:39.754 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:39.754 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:39.755 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 msgr2=0x7fe9e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:39.755 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 0x7fe9e8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe9f80096a0 tx=0x7fe9f8006b40).stop 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 msgr2=0x7fe9fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe9ec013510 tx=0x7fe9ec00b040).stop 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 shutdown_connections 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fe9e802ff60 0x7fe9e8032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 --2- 172.21.15.115:0/2580784982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9fc0f6510 0x7fe9fc06b9a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 >> 172.21.15.115:0/2580784982 conn(0x7fe9fc0f1c70 msgr2=0x7fe9fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 shutdown_connections 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:39.753+0000 7fea03867700 1 -- 172.21.15.115:0/2580784982 wait complete. 2024-01-21T07:35:39.756 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:40.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:39.752640+0000 mon.smithi107 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.115:0/2580784982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:41.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:41.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:39.871265+0000 mgr.smithi107.adejht (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:42.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 -- 172.21.15.115:0/94942446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 msgr2=0x7fca880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:42.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 --2- 172.21.15.115:0/94942446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca880f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fca78009fb0 tx=0x7fca78009b30).stop 2024-01-21T07:35:42.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 -- 172.21.15.115:0/94942446 shutdown_connections 2024-01-21T07:35:42.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 --2- 172.21.15.115:0/94942446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca880f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 -- 172.21.15.115:0/94942446 >> 172.21.15.115:0/94942446 conn(0x7fca880f1c70 msgr2=0x7fca880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 -- 172.21.15.115:0/94942446 shutdown_connections 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.306+0000 7fca90084700 1 -- 172.21.15.115:0/94942446 wait complete. 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca90084700 1 Processor -- start 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca90084700 1 -- start start 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca90084700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca90084700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88106c20 con 0x7fca880f6510 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca8de20700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca8de20700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:46560/0 (socket says 172.21.15.115:46560) 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca8de20700 1 -- 172.21.15.115:0/404061933 learned_addr learned my addr 172.21.15.115:0/404061933 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.307+0000 7fca8de20700 1 -- 172.21.15.115:0/404061933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78005040 con 0x7fca880f6510 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca8de20700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fca78013510 tx=0x7fca7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca78009320 con 0x7fca880f6510 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca88106e20 con 0x7fca880f6510 2024-01-21T07:35:42.311 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca78013f00 con 0x7fca880f6510 2024-01-21T07:35:42.312 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca78014d90 con 0x7fca880f6510 2024-01-21T07:35:42.312 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.308+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca881072b0 con 0x7fca880f6510 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca780095f0 con 0x7fca880f6510 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca88045ad0 con 0x7fca880f6510 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca7e7fc700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 0x7fca74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca78040fc0 con 0x7fca880f6510 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca8d61f700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 0x7fca74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:42.313 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.309+0000 7fca8d61f700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 0x7fca74032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fca840096a0 tx=0x7fca84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:42.315 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.314+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca78014420 con 0x7fca880f6510 2024-01-21T07:35:42.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.522+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca880fbf60 con 0x7fca880f6510 2024-01-21T07:35:42.524 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.523+0000 7fca7e7fc700 1 -- 172.21.15.115:0/404061933 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca78010230 con 0x7fca880f6510 2024-01-21T07:35:42.524 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:42.524 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:42.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 msgr2=0x7fca74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:42.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 0x7fca74032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fca840096a0 tx=0x7fca84006b40).stop 2024-01-21T07:35:42.525 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 msgr2=0x7fca88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fca78013510 tx=0x7fca7800b040).stop 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 shutdown_connections 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca7402ff60 0x7fca74032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 --2- 172.21.15.115:0/404061933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca880f6510 0x7fca88106730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 >> 172.21.15.115:0/404061933 conn(0x7fca880f1c70 msgr2=0x7fca880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 shutdown_connections 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:42.524+0000 7fca90084700 1 -- 172.21.15.115:0/404061933 wait complete. 2024-01-21T07:35:42.526 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:41.871580+0000 mgr.smithi107.adejht (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:42.523101+0000 mon.smithi107 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.115:0/404061933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:43.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:43.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:44.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:43.871875+0000 mgr.smithi107.adejht (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 -- 172.21.15.115:0/39379972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 msgr2=0x7f85f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 --2- 172.21.15.115:0/39379972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f80f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f85e8009fb0 tx=0x7f85e8009b30).stop 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 -- 172.21.15.115:0/39379972 shutdown_connections 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 --2- 172.21.15.115:0/39379972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f80f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 -- 172.21.15.115:0/39379972 >> 172.21.15.115:0/39379972 conn(0x7f85f80f0b70 msgr2=0x7f85f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 -- 172.21.15.115:0/39379972 shutdown_connections 2024-01-21T07:35:45.091 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 -- 172.21.15.115:0/39379972 wait complete. 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.090+0000 7f85fe092700 1 Processor -- start 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85fe092700 1 -- start start 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85fe092700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85fe092700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8060240 con 0x7f85f80f5410 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85f77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:45.093 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85f77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:46572/0 (socket says 172.21.15.115:46572) 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85f77fe700 1 -- 172.21.15.115:0/1306147231 learned_addr learned my addr 172.21.15.115:0/1306147231 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.091+0000 7f85f77fe700 1 -- 172.21.15.115:0/1306147231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8005040 con 0x7f85f80f5410 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85f77fe700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f85e8013510 tx=0x7f85e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e8009320 con 0x7f85f80f5410 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f8060440 con 0x7f85f80f5410 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85e8013f00 con 0x7f85f80f5410 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e8014d90 con 0x7f85f80f5410 2024-01-21T07:35:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.092+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f8109ae0 con 0x7f85f80f5410 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.093+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85e80095f0 con 0x7f85f80f5410 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.093+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f8045ad0 con 0x7f85f80f5410 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.093+0000 7f85dffff700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.093+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85e8041000 con 0x7f85f80f5410 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.093+0000 7f85f6ffd700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:45.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.094+0000 7f85f6ffd700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 0x7f85e0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:45.099 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.098+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85e8014420 con 0x7f85f80f5410 2024-01-21T07:35:45.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.301+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85f80fae60 con 0x7f85f80f5410 2024-01-21T07:35:45.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.302+0000 7f85dffff700 1 -- 172.21.15.115:0/1306147231 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85e801d7f0 con 0x7f85f80f5410 2024-01-21T07:35:45.303 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:45.303 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 msgr2=0x7f85e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 0x7f85e0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).stop 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 msgr2=0x7f85f8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f85e8013510 tx=0x7f85e800b040).stop 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 shutdown_connections 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f85e002ff60 0x7f85e0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 --2- 172.21.15.115:0/1306147231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f80f5410 0x7f85f8109430 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.303+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 >> 172.21.15.115:0/1306147231 conn(0x7f85f80f0b70 msgr2=0x7f85f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:45.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.304+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 shutdown_connections 2024-01-21T07:35:45.307 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:45.304+0000 7f85fe092700 1 -- 172.21.15.115:0/1306147231 wait complete. 2024-01-21T07:35:45.307 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:45.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:45.302449+0000 mon.smithi107 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.115:0/1306147231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:46.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:46.690 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:46.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:45.872214+0000 mgr.smithi107.adejht (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:47.909 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 -- 172.21.15.115:0/278111338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 msgr2=0x7fdd300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:47.909 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 --2- 172.21.15.115:0/278111338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd300f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fdd20009fb0 tx=0x7fdd20009b30).stop 2024-01-21T07:35:47.909 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 -- 172.21.15.115:0/278111338 shutdown_connections 2024-01-21T07:35:47.909 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 --2- 172.21.15.115:0/278111338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd300f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:47.909 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 -- 172.21.15.115:0/278111338 >> 172.21.15.115:0/278111338 conn(0x7fdd300f0b70 msgr2=0x7fdd300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 -- 172.21.15.115:0/278111338 shutdown_connections 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.906+0000 7fdd37295700 1 -- 172.21.15.115:0/278111338 wait complete. 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd37295700 1 Processor -- start 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd37295700 1 -- start start 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd37295700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd37295700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30107240 con 0x7fdd300f5410 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd35031700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd35031700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:46584/0 (socket says 172.21.15.115:46584) 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.907+0000 7fdd35031700 1 -- 172.21.15.115:0/1335627152 learned_addr learned my addr 172.21.15.115:0/1335627152 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd35031700 1 -- 172.21.15.115:0/1335627152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20005040 con 0x7fdd300f5410 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd35031700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fdd20013510 tx=0x7fdd2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd20009320 con 0x7fdd300f5410 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd30060220 con 0x7fdd300f5410 2024-01-21T07:35:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd20013f00 con 0x7fdd300f5410 2024-01-21T07:35:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd20014d90 con 0x7fdd300f5410 2024-01-21T07:35:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.908+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3010bd40 con 0x7fdd300f5410 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.909+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd200095f0 con 0x7fdd300f5410 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.909+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd30045ad0 con 0x7fdd300f5410 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.909+0000 7fdd25ffb700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 0x7fdd1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.909+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd20041000 con 0x7fdd300f5410 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.910+0000 7fdd34830700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 0x7fdd1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.910+0000 7fdd34830700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 0x7fdd1c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fdd2c0096a0 tx=0x7fdd2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:47.915 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:47.914+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd20014420 con 0x7fdd300f5410 2024-01-21T07:35:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.118+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd300fae60 con 0x7fdd300f5410 2024-01-21T07:35:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.119+0000 7fdd25ffb700 1 -- 172.21.15.115:0/1335627152 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd2001d7f0 con 0x7fdd300f5410 2024-01-21T07:35:48.120 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:48.120 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.120+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 msgr2=0x7fdd1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.120+0000 7fdd37295700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 0x7fdd1c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fdd2c0096a0 tx=0x7fdd2c006b40).stop 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.120+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 msgr2=0x7fdd3010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.120+0000 7fdd37295700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fdd20013510 tx=0x7fdd2000b040).stop 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 shutdown_connections 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdd1c02ff60 0x7fdd1c032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 --2- 172.21.15.115:0/1335627152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdd300f5410 0x7fdd3010b850 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 >> 172.21.15.115:0/1335627152 conn(0x7fdd300f0b70 msgr2=0x7fdd300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 shutdown_connections 2024-01-21T07:35:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:48.121+0000 7fdd37295700 1 -- 172.21.15.115:0/1335627152 wait complete. 2024-01-21T07:35:48.123 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:47.872490+0000 mgr.smithi107.adejht (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:48.119567+0000 mon.smithi107 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.115:0/1335627152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:49.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:49.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:50.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:49.872855+0000 mgr.smithi107.adejht (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:50.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- 172.21.15.115:0/637543740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 msgr2=0x7f26100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:50.608 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 --2- 172.21.15.115:0/637543740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26100f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f25f8009fb0 tx=0x7f25f8009b30).stop 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- 172.21.15.115:0/637543740 shutdown_connections 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 --2- 172.21.15.115:0/637543740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26100f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- 172.21.15.115:0/637543740 >> 172.21.15.115:0/637543740 conn(0x7f26100f1c70 msgr2=0x7f26100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- 172.21.15.115:0/637543740 shutdown_connections 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- 172.21.15.115:0/637543740 wait complete. 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 Processor -- start 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- start start 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.606+0000 7f2614c49700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610106bc0 con 0x7f26100f6510 2024-01-21T07:35:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f260e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f260e59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59512/0 (socket says 172.21.15.115:59512) 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f260e59c700 1 -- 172.21.15.115:0/1164998198 learned_addr learned my addr 172.21.15.115:0/1164998198 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f260e59c700 1 -- 172.21.15.115:0/1164998198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8005040 con 0x7f26100f6510 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f260e59c700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8009320 con 0x7f26100f6510 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25f8013f00 con 0x7f26100f6510 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2610106dc0 con 0x7f26100f6510 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8014d90 con 0x7f26100f6510 2024-01-21T07:35:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.607+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2610107250 con 0x7f26100f6510 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.608+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25f80095f0 con 0x7f26100f6510 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.608+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2610045ad0 con 0x7f26100f6510 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.608+0000 7f2606ffd700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 0x7f25fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.608+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25f80410c0 con 0x7f26100f6510 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.608+0000 7f260dd9b700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 0x7f25fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.609+0000 7f260dd9b700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 0x7f25fc032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f26000096a0 tx=0x7f2600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:50.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.613+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25f8014420 con 0x7f26100f6510 2024-01-21T07:35:50.820 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.819+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2610108100 con 0x7f26100f6510 2024-01-21T07:35:50.821 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.820+0000 7f2606ffd700 1 -- 172.21.15.115:0/1164998198 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f25f801d7f0 con 0x7f26100f6510 2024-01-21T07:35:50.821 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:50.821 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:50.822 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 msgr2=0x7f25fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:50.822 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 0x7f25fc032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f26000096a0 tx=0x7f2600006b40).stop 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 msgr2=0x7f26101066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).stop 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 shutdown_connections 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f25fc02ff60 0x7f25fc032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 --2- 172.21.15.115:0/1164998198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26100f6510 0x7f26101066d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 >> 172.21.15.115:0/1164998198 conn(0x7f26100f1c70 msgr2=0x7f26100f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 shutdown_connections 2024-01-21T07:35:50.823 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:50.821+0000 7f2614c49700 1 -- 172.21.15.115:0/1164998198 wait complete. 2024-01-21T07:35:50.824 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:51.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:50.820022+0000 mon.smithi107 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.115:0/1164998198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:52.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:52.218 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:53.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:51.873173+0000 mgr.smithi107.adejht (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:53.414 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 -- 172.21.15.115:0/3136315124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 msgr2=0x7f95e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 --2- 172.21.15.115:0/3136315124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e80f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f95dc009fb0 tx=0x7f95dc009b30).stop 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 -- 172.21.15.115:0/3136315124 shutdown_connections 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 --2- 172.21.15.115:0/3136315124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e80f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 -- 172.21.15.115:0/3136315124 >> 172.21.15.115:0/3136315124 conn(0x7f95e80f0b70 msgr2=0x7f95e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 -- 172.21.15.115:0/3136315124 shutdown_connections 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.412+0000 7f95ee245700 1 -- 172.21.15.115:0/3136315124 wait complete. 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95ee245700 1 Processor -- start 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95ee245700 1 -- start start 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95ee245700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95ee245700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e8060220 con 0x7f95e80f5410 2024-01-21T07:35:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:53.416 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95e77fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59524/0 (socket says 172.21.15.115:59524) 2024-01-21T07:35:53.416 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.413+0000 7f95e77fe700 1 -- 172.21.15.115:0/4208877986 learned_addr learned my addr 172.21.15.115:0/4208877986 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:53.416 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95e77fe700 1 -- 172.21.15.115:0/4208877986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc005040 con 0x7f95e80f5410 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95e77fe700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f95dc013510 tx=0x7f95dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95dc009320 con 0x7f95e80f5410 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e8060420 con 0x7f95e80f5410 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95dc013f00 con 0x7f95e80f5410 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95dc014d90 con 0x7f95e80f5410 2024-01-21T07:35:53.417 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.414+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e8109b00 con 0x7f95e80f5410 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.415+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95dc014420 con 0x7f95e80f5410 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.415+0000 7f95cffff700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 0x7f95d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.415+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95dc0410c0 con 0x7f95e80f5410 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.415+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e8059100 con 0x7f95e80f5410 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.415+0000 7f95e6ffd700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 0x7f95d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:53.418 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.416+0000 7f95e6ffd700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 0x7f95d0032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f95d80096a0 tx=0x7f95d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:53.421 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.420+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95dc0095f0 con 0x7f95e80f5410 2024-01-21T07:35:53.627 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.626+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95e8045ad0 con 0x7f95e80f5410 2024-01-21T07:35:53.628 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.627+0000 7f95cffff700 1 -- 172.21.15.115:0/4208877986 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95dc01db70 con 0x7f95e80f5410 2024-01-21T07:35:53.628 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:53.628 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:53.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 msgr2=0x7f95d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:53.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 0x7f95d0032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f95d80096a0 tx=0x7f95d8006b40).stop 2024-01-21T07:35:53.629 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 msgr2=0x7f95e8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f95dc013510 tx=0x7f95dc00b040).stop 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 shutdown_connections 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95d002ff60 0x7f95d0032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 --2- 172.21.15.115:0/4208877986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95e80f5410 0x7f95e8109450 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 >> 172.21.15.115:0/4208877986 conn(0x7f95e80f0b70 msgr2=0x7f95e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 shutdown_connections 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:53.628+0000 7f95ee245700 1 -- 172.21.15.115:0/4208877986 wait complete. 2024-01-21T07:35:53.630 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:53.627119+0000 mon.smithi107 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.115:0/4208877986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:55.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:55.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:53.873504+0000 mgr.smithi107.adejht (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 -- 172.21.15.115:0/2616033540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 msgr2=0x7f33c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2616033540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c80f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f33c4009fb0 tx=0x7f33c4009b30).stop 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 -- 172.21.15.115:0/2616033540 shutdown_connections 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2616033540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c80f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 -- 172.21.15.115:0/2616033540 >> 172.21.15.115:0/2616033540 conn(0x7f33c80f09a0 msgr2=0x7f33c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 -- 172.21.15.115:0/2616033540 shutdown_connections 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 -- 172.21.15.115:0/2616033540 wait complete. 2024-01-21T07:35:56.233 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.230+0000 7f33ce15f700 1 Processor -- start 2024-01-21T07:35:56.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33ce15f700 1 -- start start 2024-01-21T07:35:56.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33ce15f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:56.234 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33ce15f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c81058a0 con 0x7f33c80f51a0 2024-01-21T07:35:56.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33cd15d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:56.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33cd15d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59542/0 (socket says 172.21.15.115:59542) 2024-01-21T07:35:56.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33cd15d700 1 -- 172.21.15.115:0/2858475573 learned_addr learned my addr 172.21.15.115:0/2858475573 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:56.235 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.231+0000 7f33cd15d700 1 -- 172.21.15.115:0/2858475573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4005040 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33cd15d700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f33c4009830 tx=0x7f33c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c4009320 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c8105aa0 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c40136c0 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c401ee90 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.232+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c8105f30 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.233+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33c401e420 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.233+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c8045ad0 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.233+0000 7f33bdffb700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 0x7f33b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.233+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33c4041080 con 0x7f33c80f51a0 2024-01-21T07:35:56.236 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.233+0000 7f33cc95c700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 0x7f33b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:56.237 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.234+0000 7f33cc95c700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 0x7f33b40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:56.240 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.238+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c406d2b0 con 0x7f33c80f51a0 2024-01-21T07:35:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:55.873772+0000 mgr.smithi107.adejht (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:56.445 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.443+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33c8106de0 con 0x7f33c80f51a0 2024-01-21T07:35:56.446 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.444+0000 7f33bdffb700 1 -- 172.21.15.115:0/2858475573 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33c402c2d0 con 0x7f33c80f51a0 2024-01-21T07:35:56.446 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:56.446 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 msgr2=0x7f33b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 0x7f33b40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).stop 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 msgr2=0x7f33c81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f33c4009830 tx=0x7f33c400b040).stop 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 shutdown_connections 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33b402ff10 0x7f33b40323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:56.448 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 --2- 172.21.15.115:0/2858475573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33c80f51a0 0x7f33c81053b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:56.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 >> 172.21.15.115:0/2858475573 conn(0x7f33c80f09a0 msgr2=0x7f33c80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:56.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.445+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 shutdown_connections 2024-01-21T07:35:56.449 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:56.446+0000 7f33ce15f700 1 -- 172.21.15.115:0/2858475573 wait complete. 2024-01-21T07:35:56.449 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:57.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:56 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:56.445304+0000 mon.smithi107 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.115:0/2858475573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:35:57.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:35:57.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:35:58.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:57.874097+0000 mgr.smithi107.adejht (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 -- 172.21.15.115:0/636275809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 msgr2=0x7fa8080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 --2- 172.21.15.115:0/636275809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8080f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa7f8009fb0 tx=0x7fa7f8009b30).stop 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 -- 172.21.15.115:0/636275809 shutdown_connections 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 --2- 172.21.15.115:0/636275809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8080f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 -- 172.21.15.115:0/636275809 >> 172.21.15.115:0/636275809 conn(0x7fa8080f0b70 msgr2=0x7fa8080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:59.002 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 -- 172.21.15.115:0/636275809 shutdown_connections 2024-01-21T07:35:59.003 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:58.999+0000 7fa80e598700 1 -- 172.21.15.115:0/636275809 wait complete. 2024-01-21T07:35:59.003 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa80e598700 1 Processor -- start 2024-01-21T07:35:59.003 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa80e598700 1 -- start start 2024-01-21T07:35:59.003 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa80e598700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:59.003 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa80e598700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808060280 con 0x7fa8080f5410 2024-01-21T07:35:59.004 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa807fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:59.004 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa807fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59566/0 (socket says 172.21.15.115:59566) 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.000+0000 7fa807fff700 1 -- 172.21.15.115:0/2904883695 learned_addr learned my addr 172.21.15.115:0/2904883695 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa807fff700 1 -- 172.21.15.115:0/2904883695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8005040 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa807fff700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa7f8013510 tx=0x7fa7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f8009320 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7f8013f00 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f8014d90 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa808060480 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.001+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808109af0 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.002+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7f80095f0 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.002+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa808045ad0 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.002+0000 7fa804ff9700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 0x7fa7f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.002+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7f8040fc0 con 0x7fa8080f5410 2024-01-21T07:35:59.005 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.002+0000 7fa8077fe700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 0x7fa7f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:35:59.006 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.003+0000 7fa8077fe700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 0x7fa7f0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:35:59.009 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.007+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7f8014420 con 0x7fa8080f5410 2024-01-21T07:35:59.215 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.213+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa80810a310 con 0x7fa8080f5410 2024-01-21T07:35:59.216 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.214+0000 7fa804ff9700 1 -- 172.21.15.115:0/2904883695 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa7f8010230 con 0x7fa8080f5410 2024-01-21T07:35:59.216 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:35:59.216 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:35:59.217 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 msgr2=0x7fa7f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:59.217 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 0x7fa7f0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).stop 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 msgr2=0x7fa8081093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa7f8013510 tx=0x7fa7f800b040).stop 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 shutdown_connections 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa7f002ff60 0x7fa7f0032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 --2- 172.21.15.115:0/2904883695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8080f5410 0x7fa8081093c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 >> 172.21.15.115:0/2904883695 conn(0x7fa8080f0b70 msgr2=0x7fa8080f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 shutdown_connections 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:35:59.215+0000 7fa80e598700 1 -- 172.21.15.115:0/2904883695 wait complete. 2024-01-21T07:35:59.218 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:35:59.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:35:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:35:59.215280+0000 mon.smithi107 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.115:0/2904883695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:00.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:35:59.874455+0000 mgr.smithi107.adejht (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:00.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:00.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:01.785 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- 172.21.15.115:0/1080260297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 msgr2=0x7f74d80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 --2- 172.21.15.115:0/1080260297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d80f90f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f74cc009fb0 tx=0x7f74cc009b30).stop 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- 172.21.15.115:0/1080260297 shutdown_connections 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 --2- 172.21.15.115:0/1080260297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d80f90f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- 172.21.15.115:0/1080260297 >> 172.21.15.115:0/1080260297 conn(0x7f74d80f0b70 msgr2=0x7f74d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- 172.21.15.115:0/1080260297 shutdown_connections 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- 172.21.15.115:0/1080260297 wait complete. 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 Processor -- start 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.782+0000 7f74dea74700 1 -- start start 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dea74700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dea74700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d8106bd0 con 0x7f74d8060280 2024-01-21T07:36:01.786 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dc810700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dc810700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60044/0 (socket says 172.21.15.115:60044) 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dc810700 1 -- 172.21.15.115:0/3130019152 learned_addr learned my addr 172.21.15.115:0/3130019152 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dc810700 1 -- 172.21.15.115:0/3130019152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc005040 con 0x7f74d8060280 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dc810700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f74cc013510 tx=0x7f74cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc009320 con 0x7f74d8060280 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d8106dd0 con 0x7f74d8060280 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74cc013f00 con 0x7f74d8060280 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc014dc0 con 0x7f74d8060280 2024-01-21T07:36:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.783+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d8107260 con 0x7f74d8060280 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.784+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74cc014420 con 0x7f74d8060280 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.784+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d8045ad0 con 0x7f74d8060280 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.784+0000 7f74d57fa700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 0x7f74c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.785+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74cc041050 con 0x7f74d8060280 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.785+0000 7f74d7fff700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 0x7f74c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:01.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.785+0000 7f74d7fff700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 0x7f74c8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f74c40096a0 tx=0x7f74c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:01.791 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.789+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74cc0095f0 con 0x7f74d8060280 2024-01-21T07:36:01.995 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.993+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74d8108110 con 0x7f74d8060280 2024-01-21T07:36:01.996 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.994+0000 7f74d57fa700 1 -- 172.21.15.115:0/3130019152 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f74cc01cb10 con 0x7f74d8060280 2024-01-21T07:36:01.996 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:01.996 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 msgr2=0x7f74c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 0x7f74c8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f74c40096a0 tx=0x7f74c4006b40).stop 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 msgr2=0x7f74d81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f74cc013510 tx=0x7f74cc00b040).stop 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 shutdown_connections 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f74c802ff60 0x7f74c8032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 --2- 172.21.15.115:0/3130019152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74d8060280 0x7f74d81066e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 >> 172.21.15.115:0/3130019152 conn(0x7f74d80f0b70 msgr2=0x7f74d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 shutdown_connections 2024-01-21T07:36:01.998 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:01.996+0000 7f74dea74700 1 -- 172.21.15.115:0/3130019152 wait complete. 2024-01-21T07:36:01.999 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:03.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:01.874881+0000 mgr.smithi107.adejht (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:03.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:01.995429+0000 mon.smithi107 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.115:0/3130019152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:03.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:03.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:03.875290+0000 mgr.smithi107.adejht (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 -- 172.21.15.115:0/930915775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 msgr2=0x7f9bec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/930915775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec0f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9bdc009fb0 tx=0x7f9bdc009b30).stop 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 -- 172.21.15.115:0/930915775 shutdown_connections 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/930915775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec0f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 -- 172.21.15.115:0/930915775 >> 172.21.15.115:0/930915775 conn(0x7f9bec0f1c70 msgr2=0x7f9bec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 -- 172.21.15.115:0/930915775 shutdown_connections 2024-01-21T07:36:04.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.543+0000 7f9bf3b15700 1 -- 172.21.15.115:0/930915775 wait complete. 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf3b15700 1 Processor -- start 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf3b15700 1 -- start start 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf3b15700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf3b15700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec069e20 con 0x7f9bec0f6510 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf18b1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf18b1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60062/0 (socket says 172.21.15.115:60062) 2024-01-21T07:36:04.548 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf18b1700 1 -- 172.21.15.115:0/3250334794 learned_addr learned my addr 172.21.15.115:0/3250334794 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.544+0000 7f9bf18b1700 1 -- 172.21.15.115:0/3250334794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc005040 con 0x7f9bec0f6510 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9bf18b1700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9bdc013510 tx=0x7f9bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bdc009320 con 0x7f9bec0f6510 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec069fc0 con 0x7f9bec0f6510 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bdc013f00 con 0x7f9bec0f6510 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bdc014d90 con 0x7f9bec0f6510 2024-01-21T07:36:04.549 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec06a450 con 0x7f9bec0f6510 2024-01-21T07:36:04.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.545+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bdc014420 con 0x7f9bec0f6510 2024-01-21T07:36:04.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.546+0000 7f9be27fc700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 0x7f9bd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:04.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.546+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec059100 con 0x7f9bec0f6510 2024-01-21T07:36:04.550 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.546+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bdc041000 con 0x7f9bec0f6510 2024-01-21T07:36:04.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.546+0000 7f9bf10b0700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 0x7f9bd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:04.551 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.546+0000 7f9bf10b0700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 0x7f9bd8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9be80096a0 tx=0x7f9be8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:04.552 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.551+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bdc0095f0 con 0x7f9bec0f6510 2024-01-21T07:36:04.760 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.758+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bec0fbf60 con 0x7f9bec0f6510 2024-01-21T07:36:04.760 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.758+0000 7f9be27fc700 1 -- 172.21.15.115:0/3250334794 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9bdc02c090 con 0x7f9bec0f6510 2024-01-21T07:36:04.761 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:04.761 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 msgr2=0x7f9bd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 0x7f9bd8032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9be80096a0 tx=0x7f9be8006b40).stop 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 msgr2=0x7f9bec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9bdc013510 tx=0x7f9bdc00b040).stop 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 shutdown_connections 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9bd802ff60 0x7f9bd8032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 --2- 172.21.15.115:0/3250334794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bec0f6510 0x7f9bec06b9a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 >> 172.21.15.115:0/3250334794 conn(0x7f9bec0f1c70 msgr2=0x7f9bec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 shutdown_connections 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:04.760+0000 7f9bf3b15700 1 -- 172.21.15.115:0/3250334794 wait complete. 2024-01-21T07:36:04.763 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:04.759813+0000 mon.smithi107 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.115:0/3250334794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:06.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:06.150 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:05.875623+0000 mgr.smithi107.adejht (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:07.358 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 -- 172.21.15.115:0/2378742081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 msgr2=0x7fbcfc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2378742081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc0f66c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fbcf0009fb0 tx=0x7fbcf0009b30).stop 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 -- 172.21.15.115:0/2378742081 shutdown_connections 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2378742081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc0f66c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 -- 172.21.15.115:0/2378742081 >> 172.21.15.115:0/2378742081 conn(0x7fbcfc0f1aa0 msgr2=0x7fbcfc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 -- 172.21.15.115:0/2378742081 shutdown_connections 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.355+0000 7fbd00e05700 1 -- 172.21.15.115:0/2378742081 wait complete. 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbd00e05700 1 Processor -- start 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbd00e05700 1 -- start start 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbd00e05700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbd00e05700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc1069a0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbcfb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbcfb7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:60074/0 (socket says 172.21.15.115:60074) 2024-01-21T07:36:07.359 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.356+0000 7fbcfb7fe700 1 -- 172.21.15.115:0/2097738493 learned_addr learned my addr 172.21.15.115:0/2097738493 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:07.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbcfb7fe700 1 -- 172.21.15.115:0/2097738493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf0005040 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbcfb7fe700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fbcf0013510 tx=0x7fbcf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:07.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0009320 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcf0013f00 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.361 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0014dc0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc106ba0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.357+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc107030 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.358+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcf001d3d0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.358+0000 7fbce3fff700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 0x7fbce4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.358+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcf00413e0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.358+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc045ad0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.358+0000 7fbcfaffd700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 0x7fbce4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:07.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.359+0000 7fbcfaffd700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 0x7fbce4032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbcec0096a0 tx=0x7fbcec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:07.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.363+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcf0012920 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.567 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.565+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcfc0fbdd0 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.568 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.566+0000 7fbce3fff700 1 -- 172.21.15.115:0/2097738493 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcf002b030 con 0x7fbcfc0f62a0 2024-01-21T07:36:07.568 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:07.568 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 msgr2=0x7fbce4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 0x7fbce4032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbcec0096a0 tx=0x7fbcec006b40).stop 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 msgr2=0x7fbcfc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fbcf0013510 tx=0x7fbcf000b040).stop 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 shutdown_connections 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbce402ffb0 0x7fbce4032470 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 --2- 172.21.15.115:0/2097738493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcfc0f62a0 0x7fbcfc1064b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:07.570 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 >> 172.21.15.115:0/2097738493 conn(0x7fbcfc0f1aa0 msgr2=0x7fbcfc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:07.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 shutdown_connections 2024-01-21T07:36:07.571 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:07.568+0000 7fbd00e05700 1 -- 172.21.15.115:0/2097738493 wait complete. 2024-01-21T07:36:07.571 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:07.567381+0000 mon.smithi107 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.115:0/2097738493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:08.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:08.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:09.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:08 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:07.876016+0000 mgr.smithi107.adejht (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:10.094 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 -- 172.21.15.115:0/3893250861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 msgr2=0x7f52dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 --2- 172.21.15.115:0/3893250861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f52d0009fb0 tx=0x7f52d0009b30).stop 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 -- 172.21.15.115:0/3893250861 shutdown_connections 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 --2- 172.21.15.115:0/3893250861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 -- 172.21.15.115:0/3893250861 >> 172.21.15.115:0/3893250861 conn(0x7f52dc0f0b70 msgr2=0x7f52dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 -- 172.21.15.115:0/3893250861 shutdown_connections 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.091+0000 7f52e2d53700 1 -- 172.21.15.115:0/3893250861 wait complete. 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e2d53700 1 Processor -- start 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e2d53700 1 -- start start 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e2d53700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e2d53700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0fb820 con 0x7f52dc0f5410 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e0aef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e0aef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47016/0 (socket says 172.21.15.115:47016) 2024-01-21T07:36:10.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.092+0000 7f52e0aef700 1 -- 172.21.15.115:0/1933458942 learned_addr learned my addr 172.21.15.115:0/1933458942 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:10.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52e0aef700 1 -- 172.21.15.115:0/1933458942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0005040 con 0x7f52dc0f5410 2024-01-21T07:36:10.096 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52e0aef700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f52d0013510 tx=0x7f52d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d0009320 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc0fb9c0 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52d0013f00 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d0014d90 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.093+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc0fbe50 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52d00095f0 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc059100 con 0x7f52dc0f5410 2024-01-21T07:36:10.097 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52d97fa700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 0x7f52cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:10.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52d00410c0 con 0x7f52dc0f5410 2024-01-21T07:36:10.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52dbfff700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 0x7f52cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:10.098 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.094+0000 7f52dbfff700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 0x7f52cc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f52c80096a0 tx=0x7f52c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:10.101 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.099+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52d0014420 con 0x7f52dc0f5410 2024-01-21T07:36:10.307 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.305+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52dc0fae60 con 0x7f52dc0f5410 2024-01-21T07:36:10.308 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.306+0000 7f52d97fa700 1 -- 172.21.15.115:0/1933458942 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f52d001d7f0 con 0x7f52dc0f5410 2024-01-21T07:36:10.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:10.308 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 msgr2=0x7f52cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 0x7f52cc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f52c80096a0 tx=0x7f52c8006b40).stop 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 msgr2=0x7f52dc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f52d0013510 tx=0x7f52d000b040).stop 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 shutdown_connections 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f52cc02ff60 0x7f52cc032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.307+0000 7f52e2d53700 1 --2- 172.21.15.115:0/1933458942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc0f5410 0x7f52dc0fd3a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.308+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 >> 172.21.15.115:0/1933458942 conn(0x7f52dc0f0b70 msgr2=0x7f52dc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.308+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 shutdown_connections 2024-01-21T07:36:10.310 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:10.308+0000 7f52e2d53700 1 -- 172.21.15.115:0/1933458942 wait complete. 2024-01-21T07:36:10.311 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:09.876326+0000 mgr.smithi107.adejht (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:09.951863+0000 mon.smithi107 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:36:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:09.952072+0000 mon.smithi107 (mon.0) 280 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:36:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:10.307402+0000 mon.smithi107 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.115:0/1933458942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:11.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:11.678 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 -- 172.21.15.115:0/267150894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 msgr2=0x7ff74c0f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 --2- 172.21.15.115:0/267150894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c0f66e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7ff740009fb0 tx=0x7ff740009b30).stop 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 -- 172.21.15.115:0/267150894 shutdown_connections 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 --2- 172.21.15.115:0/267150894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c0f66e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 -- 172.21.15.115:0/267150894 >> 172.21.15.115:0/267150894 conn(0x7ff74c0f1ac0 msgr2=0x7ff74c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 -- 172.21.15.115:0/267150894 shutdown_connections 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 -- 172.21.15.115:0/267150894 wait complete. 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.791+0000 7ff751dfe700 1 Processor -- start 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff751dfe700 1 -- start start 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff751dfe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:12.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff751dfe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c106a10 con 0x7ff74c0f62c0 2024-01-21T07:36:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff750dfc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff750dfc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47036/0 (socket says 172.21.15.115:47036) 2024-01-21T07:36:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff750dfc700 1 -- 172.21.15.115:0/3917762934 learned_addr learned my addr 172.21.15.115:0/3917762934 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.792+0000 7ff750dfc700 1 -- 172.21.15.115:0/3917762934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740005040 con 0x7ff74c0f62c0 2024-01-21T07:36:12.796 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff750dfc700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff740009830 tx=0x7ff74000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff740009320 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74c106c10 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7400136c0 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74001ee90 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c1070a0 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.793+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff74001e420 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.794+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c045ad0 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.794+0000 7ff7497fa700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 0x7ff73c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.794+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff740040fc0 con 0x7ff74c0f62c0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.794+0000 7ff74bfff700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 0x7ff73c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:12.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.794+0000 7ff74bfff700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 0x7ff73c0323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff7380096a0 tx=0x7ff738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:12.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:12.799+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74006d2b0 con 0x7ff74c0f62c0 2024-01-21T07:36:13.007 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.005+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff74c107f50 con 0x7ff74c0f62c0 2024-01-21T07:36:13.008 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.006+0000 7ff7497fa700 1 -- 172.21.15.115:0/3917762934 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff7400095f0 con 0x7ff74c0f62c0 2024-01-21T07:36:13.008 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:13.008 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.007+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 msgr2=0x7ff73c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.007+0000 7ff751dfe700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 0x7ff73c0323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff7380096a0 tx=0x7ff738006b40).stop 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.007+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 msgr2=0x7ff74c106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.007+0000 7ff751dfe700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff740009830 tx=0x7ff74000b040).stop 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 shutdown_connections 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff73c02ff10 0x7ff73c0323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 --2- 172.21.15.115:0/3917762934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff74c0f62c0 0x7ff74c106520 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 >> 172.21.15.115:0/3917762934 conn(0x7ff74c0f1ac0 msgr2=0x7ff74c0f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 shutdown_connections 2024-01-21T07:36:13.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:13.008+0000 7ff751dfe700 1 -- 172.21.15.115:0/3917762934 wait complete. 2024-01-21T07:36:13.011 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:13.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:11.876666+0000 mgr.smithi107.adejht (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:14.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:13.007411+0000 mon.smithi107 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.115:0/3917762934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:14.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:14.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:15.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:13.877036+0000 mgr.smithi107.adejht (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- 172.21.15.115:0/1841994852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 msgr2=0x7fcb5c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 --2- 172.21.15.115:0/1841994852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f9190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fcb4c009fb0 tx=0x7fcb4c009b30).stop 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- 172.21.15.115:0/1841994852 shutdown_connections 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 --2- 172.21.15.115:0/1841994852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f9190 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- 172.21.15.115:0/1841994852 >> 172.21.15.115:0/1841994852 conn(0x7fcb5c0f0b70 msgr2=0x7fcb5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- 172.21.15.115:0/1841994852 shutdown_connections 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- 172.21.15.115:0/1841994852 wait complete. 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 Processor -- start 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- start start 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.538+0000 7fcb61c19700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0f5090 con 0x7fcb5c0601e0 2024-01-21T07:36:15.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb5b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb5b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47046/0 (socket says 172.21.15.115:47046) 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb5b7fe700 1 -- 172.21.15.115:0/2824787107 learned_addr learned my addr 172.21.15.115:0/2824787107 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb5b7fe700 1 -- 172.21.15.115:0/2824787107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c005040 con 0x7fcb5c0601e0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb5b7fe700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcb4c004dd0 tx=0x7fcb4c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c018070 con 0x7fcb5c0601e0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c0f5230 con 0x7fcb5c0601e0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb4c013410 con 0x7fcb5c0601e0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4c01c3c0 con 0x7fcb5c0601e0 2024-01-21T07:36:15.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.539+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c0f56c0 con 0x7fcb5c0601e0 2024-01-21T07:36:15.544 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.540+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb4c01d420 con 0x7fcb5c0601e0 2024-01-21T07:36:15.544 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.540+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c059100 con 0x7fcb5c0601e0 2024-01-21T07:36:15.544 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.540+0000 7fcb43fff700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 0x7fcb44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:15.544 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.540+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb4c0413d0 con 0x7fcb5c0601e0 2024-01-21T07:36:15.544 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.541+0000 7fcb5affd700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 0x7fcb44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:15.545 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.541+0000 7fcb5affd700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 0x7fcb44032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fcb500096a0 tx=0x7fcb50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:15.547 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.545+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb4c01d820 con 0x7fcb5c0601e0 2024-01-21T07:36:15.754 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.752+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb5c0fbf60 con 0x7fcb5c0601e0 2024-01-21T07:36:15.754 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.752+0000 7fcb43fff700 1 -- 172.21.15.115:0/2824787107 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb4c01b370 con 0x7fcb5c0601e0 2024-01-21T07:36:15.755 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:15.755 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:15.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 msgr2=0x7fcb44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:15.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 0x7fcb44032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fcb500096a0 tx=0x7fcb50006b40).stop 2024-01-21T07:36:15.756 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 msgr2=0x7fcb5c0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcb4c004dd0 tx=0x7fcb4c009b30).stop 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 shutdown_connections 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcb4402ff60 0x7fcb44032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 --2- 172.21.15.115:0/2824787107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb5c0601e0 0x7fcb5c0f6c10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 >> 172.21.15.115:0/2824787107 conn(0x7fcb5c0f0b70 msgr2=0x7fcb5c0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 shutdown_connections 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:15.754+0000 7fcb61c19700 1 -- 172.21.15.115:0/2824787107 wait complete. 2024-01-21T07:36:15.757 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:15.754003+0000 mon.smithi107 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.115:0/2824787107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:17.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:17.169 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:17.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:15.877416+0000 mgr.smithi107.adejht (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 -- 172.21.15.115:0/4108592577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 msgr2=0x7f07dc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/4108592577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f07c4009fb0 tx=0x7f07c4009b30).stop 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 -- 172.21.15.115:0/4108592577 shutdown_connections 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/4108592577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 -- 172.21.15.115:0/4108592577 >> 172.21.15.115:0/4108592577 conn(0x7f07dc0f0b70 msgr2=0x7f07dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:18.292 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 -- 172.21.15.115:0/4108592577 shutdown_connections 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.288+0000 7f07e0bfd700 1 -- 172.21.15.115:0/4108592577 wait complete. 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07e0bfd700 1 Processor -- start 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07e0bfd700 1 -- start start 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07e0bfd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07e0bfd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc0f4ff0 con 0x7f07dc060280 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07da59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07da59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:47056/0 (socket says 172.21.15.115:47056) 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07da59c700 1 -- 172.21.15.115:0/1862650622 learned_addr learned my addr 172.21.15.115:0/1862650622 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.289+0000 7f07da59c700 1 -- 172.21.15.115:0/1862650622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4005040 con 0x7f07dc060280 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07da59c700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f07c4013510 tx=0x7f07c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4009320 con 0x7f07dc060280 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c4013f00 con 0x7f07dc060280 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4014dc0 con 0x7f07dc060280 2024-01-21T07:36:18.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc0f5190 con 0x7f07dc060280 2024-01-21T07:36:18.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc0f5620 con 0x7f07dc060280 2024-01-21T07:36:18.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.290+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07c40095f0 con 0x7f07dc060280 2024-01-21T07:36:18.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.291+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07dc059100 con 0x7f07dc060280 2024-01-21T07:36:18.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.291+0000 7f07d2ffd700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 0x7f07c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:18.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.291+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07c4041020 con 0x7f07dc060280 2024-01-21T07:36:18.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.291+0000 7f07d9d9b700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 0x7f07c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:18.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.291+0000 7f07d9d9b700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 0x7f07c8032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:18.298 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.296+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07c4014420 con 0x7f07dc060280 2024-01-21T07:36:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:17.877795+0000 mgr.smithi107.adejht (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:18.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.504+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07dc0fbf50 con 0x7f07dc060280 2024-01-21T07:36:18.506 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.505+0000 7f07d2ffd700 1 -- 172.21.15.115:0/1862650622 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f07c401c920 con 0x7f07dc060280 2024-01-21T07:36:18.507 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:18.507 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 msgr2=0x7f07c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 0x7f07c8032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).stop 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 msgr2=0x7f07dc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f07c4013510 tx=0x7f07c400b040).stop 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 shutdown_connections 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f07c802ff60 0x7f07c8032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 --2- 172.21.15.115:0/1862650622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07dc060280 0x7f07dc0f6b70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 >> 172.21.15.115:0/1862650622 conn(0x7f07dc0f0b70 msgr2=0x7f07dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 shutdown_connections 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:18.506+0000 7f07e0bfd700 1 -- 172.21.15.115:0/1862650622 wait complete. 2024-01-21T07:36:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:19.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:18 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:18.506025+0000 mon.smithi107 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.115:0/1862650622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:19.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:19.884 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:20.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:19.878059+0000 mgr.smithi107.adejht (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:21.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- 172.21.15.115:0/70199479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 msgr2=0x7f9e200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/70199479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e200f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9e10009fb0 tx=0x7f9e10009b30).stop 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- 172.21.15.115:0/70199479 shutdown_connections 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/70199479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e200f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- 172.21.15.115:0/70199479 >> 172.21.15.115:0/70199479 conn(0x7f9e200f09a0 msgr2=0x7f9e200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- 172.21.15.115:0/70199479 shutdown_connections 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- 172.21.15.115:0/70199479 wait complete. 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 Processor -- start 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.103+0000 7f9e1ffff700 1 -- start start 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1ffff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e201058a0 con 0x7f9e200f51a0 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:21.107 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:41534/0 (socket says 172.21.15.115:41534) 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1effd700 1 -- 172.21.15.115:0/77013634 learned_addr learned my addr 172.21.15.115:0/77013634 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1effd700 1 -- 172.21.15.115:0/77013634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10005040 con 0x7f9e200f51a0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1effd700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9e10013510 tx=0x7f9e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e10009320 con 0x7f9e200f51a0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e20105aa0 con 0x7f9e200f51a0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e10013f00 con 0x7f9e200f51a0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e20105f30 con 0x7f9e200f51a0 2024-01-21T07:36:21.108 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.104+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e10014dc0 con 0x7f9e200f51a0 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.105+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e100095f0 con 0x7f9e200f51a0 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.105+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e20045ad0 con 0x7f9e200f51a0 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.105+0000 7f9e077fe700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 0x7f9e08032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.105+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e10042d50 con 0x7f9e200f51a0 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.106+0000 7f9e1e7fc700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 0x7f9e08032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:21.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.106+0000 7f9e1e7fc700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 0x7f9e08032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9e140096a0 tx=0x7f9e14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:21.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.110+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e10014420 con 0x7f9e200f51a0 2024-01-21T07:36:21.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.315+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e20106d80 con 0x7f9e200f51a0 2024-01-21T07:36:21.317 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.315+0000 7f9e077fe700 1 -- 172.21.15.115:0/77013634 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9e10011290 con 0x7f9e200f51a0 2024-01-21T07:36:21.318 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:21.318 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:21.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 msgr2=0x7f9e08032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:21.319 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 0x7f9e08032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9e140096a0 tx=0x7f9e14006b40).stop 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 msgr2=0x7f9e201053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9e10013510 tx=0x7f9e1000b040).stop 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 shutdown_connections 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9e0802ffb0 0x7f9e08032470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 --2- 172.21.15.115:0/77013634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e200f51a0 0x7f9e201053b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 >> 172.21.15.115:0/77013634 conn(0x7f9e200f09a0 msgr2=0x7f9e200f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 shutdown_connections 2024-01-21T07:36:21.320 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:21.317+0000 7f9e1ffff700 1 -- 172.21.15.115:0/77013634 wait complete. 2024-01-21T07:36:21.321 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:21.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:21 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:21.316838+0000 mon.smithi107 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.115:0/77013634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:22.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:22.740 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:21.878416+0000 mgr.smithi107.adejht (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:23.869 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 -- 172.21.15.115:0/1731149607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 msgr2=0x7f98fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 --2- 172.21.15.115:0/1731149607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f55c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f98ec009fb0 tx=0x7f98ec009b30).stop 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 -- 172.21.15.115:0/1731149607 shutdown_connections 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 --2- 172.21.15.115:0/1731149607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f55c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 -- 172.21.15.115:0/1731149607 >> 172.21.15.115:0/1731149607 conn(0x7f98fc0f09a0 msgr2=0x7f98fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 -- 172.21.15.115:0/1731149607 shutdown_connections 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.864+0000 7f98fbfff700 1 -- 172.21.15.115:0/1731149607 wait complete. 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98fbfff700 1 Processor -- start 2024-01-21T07:36:23.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98fbfff700 1 -- start start 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98fbfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98fbfff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0f13d0 con 0x7f98fc0f51a0 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:41562/0 (socket says 172.21.15.115:41562) 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98faffd700 1 -- 172.21.15.115:0/2118633577 learned_addr learned my addr 172.21.15.115:0/2118633577 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98faffd700 1 -- 172.21.15.115:0/2118633577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec005040 con 0x7f98fc0f51a0 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.865+0000 7f98faffd700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f98ec013510 tx=0x7f98ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:23.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ec009320 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98ec013f00 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0f15d0 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ec014dc0 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc0f1a60 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.866+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98ec01d3d0 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.867+0000 7f98e37fe700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 0x7f98e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.867+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc059100 con 0x7f98fc0f51a0 2024-01-21T07:36:23.872 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.867+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98ec041300 con 0x7f98fc0f51a0 2024-01-21T07:36:23.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.867+0000 7f98fa7fc700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 0x7f98e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:23.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.867+0000 7f98fa7fc700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 0x7f98e4032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f98f00096a0 tx=0x7f98f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:23.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:23.871+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98ec012330 con 0x7f98fc0f51a0 2024-01-21T07:36:24.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.081+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98fc027a00 con 0x7f98fc0f51a0 2024-01-21T07:36:24.083 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.081+0000 7f98e37fe700 1 -- 172.21.15.115:0/2118633577 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98ec02b030 con 0x7f98fc0f51a0 2024-01-21T07:36:24.084 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:24.084 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 msgr2=0x7f98e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 0x7f98e4032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f98f00096a0 tx=0x7f98f0006b40).stop 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 msgr2=0x7f98fc0f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f98ec013510 tx=0x7f98ec00b040).stop 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 shutdown_connections 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98e402ff60 0x7f98e4032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 --2- 172.21.15.115:0/2118633577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98fc0f51a0 0x7f98fc0f2f50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:24.085 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 >> 172.21.15.115:0/2118633577 conn(0x7f98fc0f09a0 msgr2=0x7f98fc05fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:24.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 shutdown_connections 2024-01-21T07:36:24.086 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:24.083+0000 7f98fbfff700 1 -- 172.21.15.115:0/2118633577 wait complete. 2024-01-21T07:36:24.086 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:23.878779+0000 mgr.smithi107.adejht (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:24.082817+0000 mon.smithi107 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.115:0/2118633577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:25.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:25.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:25.879159+0000 mgr.smithi107.adejht (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 -- 172.21.15.115:0/3315697716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 msgr2=0x7fee740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 --2- 172.21.15.115:0/3315697716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee740f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fee68009fb0 tx=0x7fee68009b30).stop 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 -- 172.21.15.115:0/3315697716 shutdown_connections 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 --2- 172.21.15.115:0/3315697716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee740f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 -- 172.21.15.115:0/3315697716 >> 172.21.15.115:0/3315697716 conn(0x7fee740f0b70 msgr2=0x7fee740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 -- 172.21.15.115:0/3315697716 shutdown_connections 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.588+0000 7fee7883d700 1 -- 172.21.15.115:0/3315697716 wait complete. 2024-01-21T07:36:26.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7883d700 1 Processor -- start 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7883d700 1 -- start start 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7883d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7883d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74060240 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:41586/0 (socket says 172.21.15.115:41586) 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7259c700 1 -- 172.21.15.115:0/1974583000 learned_addr learned my addr 172.21.15.115:0/1974583000 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.589+0000 7fee7259c700 1 -- 172.21.15.115:0/1974583000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee68005040 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee7259c700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fee68013510 tx=0x7fee6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee68009320 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74060440 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee68013f00 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee68014d90 con 0x7fee740f5410 2024-01-21T07:36:26.593 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee74109ae0 con 0x7fee740f5410 2024-01-21T07:36:26.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.590+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee680095f0 con 0x7fee740f5410 2024-01-21T07:36:26.594 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.591+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee74045ad0 con 0x7fee740f5410 2024-01-21T07:36:26.595 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.591+0000 7fee62ffd700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:26.595 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.591+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee68040fc0 con 0x7fee740f5410 2024-01-21T07:36:26.595 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.591+0000 7fee71d9b700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:26.595 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.591+0000 7fee71d9b700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 0x7fee5c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fee640096a0 tx=0x7fee64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:26.597 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.595+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee68014420 con 0x7fee740f5410 2024-01-21T07:36:26.806 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.804+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee740fae60 con 0x7fee740f5410 2024-01-21T07:36:26.807 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.805+0000 7fee62ffd700 1 -- 172.21.15.115:0/1974583000 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee68010230 con 0x7fee740f5410 2024-01-21T07:36:26.807 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:26.807 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 msgr2=0x7fee5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 0x7fee5c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fee640096a0 tx=0x7fee64006b40).stop 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 msgr2=0x7fee74109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fee68013510 tx=0x7fee6800b040).stop 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 shutdown_connections 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fee5c02ff60 0x7fee5c032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 --2- 172.21.15.115:0/1974583000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee740f5410 0x7fee74109430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 >> 172.21.15.115:0/1974583000 conn(0x7fee740f0b70 msgr2=0x7fee740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:26.809 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 shutdown_connections 2024-01-21T07:36:26.810 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:26.806+0000 7fee7883d700 1 -- 172.21.15.115:0/1974583000 wait complete. 2024-01-21T07:36:26.810 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:27.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:26 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:26.806054+0000 mon.smithi107 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.115:0/1974583000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:28.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:28.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:27.879581+0000 mgr.smithi107.adejht (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/1384868373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 msgr2=0x7fbd580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/1384868373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd580f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fbd48009fb0 tx=0x7fbd48009b30).stop 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/1384868373 shutdown_connections 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/1384868373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd580f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/1384868373 >> 172.21.15.115:0/1384868373 conn(0x7fbd580f0b70 msgr2=0x7fbd580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/1384868373 shutdown_connections 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.358+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/1384868373 wait complete. 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5ffd3700 1 Processor -- start 2024-01-21T07:36:29.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5ffd3700 1 -- start start 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5ffd3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5ffd3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58060220 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5dd6f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5dd6f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35080/0 (socket says 172.21.15.115:35080) 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5dd6f700 1 -- 172.21.15.115:0/3435763275 learned_addr learned my addr 172.21.15.115:0/3435763275 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5dd6f700 1 -- 172.21.15.115:0/3435763275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd48005040 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.359+0000 7fbd5dd6f700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fbd48013510 tx=0x7fbd4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd48009320 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd48013f00 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd58060420 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd48014d90 con 0x7fbd580f5410 2024-01-21T07:36:29.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd58109b00 con 0x7fbd580f5410 2024-01-21T07:36:29.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd480095f0 con 0x7fbd580f5410 2024-01-21T07:36:29.364 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.360+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd58059100 con 0x7fbd580f5410 2024-01-21T07:36:29.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.361+0000 7fbd4e7fc700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 0x7fbd44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:29.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.361+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd480410c0 con 0x7fbd580f5410 2024-01-21T07:36:29.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.361+0000 7fbd5d56e700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 0x7fbd44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:29.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.361+0000 7fbd5d56e700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 0x7fbd44032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fbd540096a0 tx=0x7fbd54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:29.367 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.366+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd4801d7f0 con 0x7fbd580f5410 2024-01-21T07:36:29.573 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.571+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd58045ad0 con 0x7fbd580f5410 2024-01-21T07:36:29.574 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd4e7fc700 1 -- 172.21.15.115:0/3435763275 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbd48010230 con 0x7fbd580f5410 2024-01-21T07:36:29.574 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:29.574 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:29.575 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 msgr2=0x7fbd44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 0x7fbd44032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fbd540096a0 tx=0x7fbd54006b40).stop 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 msgr2=0x7fbd58109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fbd48013510 tx=0x7fbd4800b040).stop 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 shutdown_connections 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fbd4402ff60 0x7fbd44032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 --2- 172.21.15.115:0/3435763275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd580f5410 0x7fbd58109450 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 >> 172.21.15.115:0/3435763275 conn(0x7fbd580f0b70 msgr2=0x7fbd580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 shutdown_connections 2024-01-21T07:36:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:29.573+0000 7fbd5ffd3700 1 -- 172.21.15.115:0/3435763275 wait complete. 2024-01-21T07:36:29.577 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:30.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:29.573129+0000 mon.smithi107 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.115:0/3435763275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:30.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:30.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:31.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:29.879951+0000 mgr.smithi107.adejht (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:32.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1068887568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 msgr2=0x7fce9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:32.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1068887568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c0f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fce98009fb0 tx=0x7fce98009b30).stop 2024-01-21T07:36:32.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1068887568 shutdown_connections 2024-01-21T07:36:32.116 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1068887568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c0f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1068887568 >> 172.21.15.115:0/1068887568 conn(0x7fce9c0f0b70 msgr2=0x7fce9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1068887568 shutdown_connections 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.113+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1068887568 wait complete. 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea3ceb700 1 Processor -- start 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea3ceb700 1 -- start start 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea3ceb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea3ceb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c105b20 con 0x7fce9c0f5410 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea1a87700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea1a87700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35094/0 (socket says 172.21.15.115:35094) 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea1a87700 1 -- 172.21.15.115:0/1353840228 learned_addr learned my addr 172.21.15.115:0/1353840228 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.114+0000 7fcea1a87700 1 -- 172.21.15.115:0/1353840228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98005040 con 0x7fce9c0f5410 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fcea1a87700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fce98014000 tx=0x7fce9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce980095f0 con 0x7fce9c0f5410 2024-01-21T07:36:32.117 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c105d20 con 0x7fce9c0f5410 2024-01-21T07:36:32.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce98007e70 con 0x7fce9c0f5410 2024-01-21T07:36:32.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce98014ea0 con 0x7fce9c0f5410 2024-01-21T07:36:32.118 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.115+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c1061b0 con 0x7fce9c0f5410 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce98014690 con 0x7fce9c0f5410 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c045ad0 con 0x7fce9c0f5410 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fce927fc700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 0x7fce88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce98042660 con 0x7fce9c0f5410 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fcea1286700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 0x7fce88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:32.119 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.116+0000 7fcea1286700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 0x7fce88032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fce8c0096a0 tx=0x7fce8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:32.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.120+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce980272f0 con 0x7fce9c0f5410 2024-01-21T07:36:32.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.326+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce9c0fae60 con 0x7fce9c0f5410 2024-01-21T07:36:32.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.327+0000 7fce927fc700 1 -- 172.21.15.115:0/1353840228 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce980272f0 con 0x7fce9c0f5410 2024-01-21T07:36:32.328 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:32.329 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 msgr2=0x7fce88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 0x7fce88032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fce8c0096a0 tx=0x7fce8c006b40).stop 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 msgr2=0x7fce9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fce98014000 tx=0x7fce9800b040).stop 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 shutdown_connections 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fce8802ff60 0x7fce88032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 --2- 172.21.15.115:0/1353840228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce9c0f5410 0x7fce9c105630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:32.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 >> 172.21.15.115:0/1353840228 conn(0x7fce9c0f0b70 msgr2=0x7fce9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:32.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 shutdown_connections 2024-01-21T07:36:32.331 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:32.329+0000 7fcea3ceb700 1 -- 172.21.15.115:0/1353840228 wait complete. 2024-01-21T07:36:32.331 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:33.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:31.880347+0000 mgr.smithi107.adejht (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:33.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:32.327756+0000 mon.smithi107 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.115:0/1353840228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:33.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:33.723 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:33.880647+0000 mgr.smithi107.adejht (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1917220632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 msgr2=0x7f90c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1917220632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c40f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f90b4009fb0 tx=0x7f90b4009b30).stop 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1917220632 shutdown_connections 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1917220632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c40f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1917220632 >> 172.21.15.115:0/1917220632 conn(0x7f90c40f0b70 msgr2=0x7f90c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1917220632 shutdown_connections 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.896+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1917220632 wait complete. 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90cc4ae700 1 Processor -- start 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90cc4ae700 1 -- start start 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90cc4ae700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90cc4ae700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4106c00 con 0x7f90c40f5410 2024-01-21T07:36:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90ca24a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90ca24a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35104/0 (socket says 172.21.15.115:35104) 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90ca24a700 1 -- 172.21.15.115:0/1681159130 learned_addr learned my addr 172.21.15.115:0/1681159130 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90ca24a700 1 -- 172.21.15.115:0/1681159130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4005040 con 0x7f90c40f5410 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.897+0000 7f90ca24a700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f90b4013510 tx=0x7f90b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90b4009320 con 0x7f90c40f5410 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90b4013f00 con 0x7f90c40f5410 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c4106e00 con 0x7f90c40f5410 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90b4014d90 con 0x7f90c40f5410 2024-01-21T07:36:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c4107290 con 0x7f90c40f5410 2024-01-21T07:36:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90b40263a0 con 0x7f90c40f5410 2024-01-21T07:36:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.898+0000 7f90baffd700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 0x7f90b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.899+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90b4041000 con 0x7f90c40f5410 2024-01-21T07:36:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.899+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c4045ad0 con 0x7f90c40f5410 2024-01-21T07:36:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.899+0000 7f90c9a49700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 0x7f90b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:34.902 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.899+0000 7f90c9a49700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 0x7f90b0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f90c00096a0 tx=0x7f90c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:34.904 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:34.903+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90b4012920 con 0x7f90c40f5410 2024-01-21T07:36:35.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.108+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90c4107fb0 con 0x7f90c40f5410 2024-01-21T07:36:35.109 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.108+0000 7f90baffd700 1 -- 172.21.15.115:0/1681159130 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f90b402b020 con 0x7f90c40f5410 2024-01-21T07:36:35.109 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:35.110 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:35.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 msgr2=0x7f90b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:35.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 0x7f90b0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f90c00096a0 tx=0x7f90c0006b40).stop 2024-01-21T07:36:35.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 msgr2=0x7f90c4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:35.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f90b4013510 tx=0x7f90b400b040).stop 2024-01-21T07:36:35.111 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 shutdown_connections 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f90b002ff60 0x7f90b0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 --2- 172.21.15.115:0/1681159130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90c40f5410 0x7f90c4106710 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 >> 172.21.15.115:0/1681159130 conn(0x7f90c40f0b70 msgr2=0x7f90c405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 shutdown_connections 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:35.110+0000 7f90cc4ae700 1 -- 172.21.15.115:0/1681159130 wait complete. 2024-01-21T07:36:35.112 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:35.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:35.108864+0000 mon.smithi107 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.115:0/1681159130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:36.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:36.503 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:36.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:35.880941+0000 mgr.smithi107.adejht (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:37.656 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 -- 172.21.15.115:0/3995943633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 msgr2=0x7f53fc0f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 --2- 172.21.15.115:0/3995943633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc0f6e30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f53ec009fb0 tx=0x7f53ec009b30).stop 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 -- 172.21.15.115:0/3995943633 shutdown_connections 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 --2- 172.21.15.115:0/3995943633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc0f6e30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 -- 172.21.15.115:0/3995943633 >> 172.21.15.115:0/3995943633 conn(0x7f53fc0f02e0 msgr2=0x7f53fc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 -- 172.21.15.115:0/3995943633 shutdown_connections 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.653+0000 7f540134c700 1 -- 172.21.15.115:0/3995943633 wait complete. 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.654+0000 7f540134c700 1 Processor -- start 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.654+0000 7f540134c700 1 -- start start 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.654+0000 7f540134c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.654+0000 7f540134c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc105ac0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:37.657 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53faffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:35130/0 (socket says 172.21.15.115:35130) 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53faffd700 1 -- 172.21.15.115:0/470068586 learned_addr learned my addr 172.21.15.115:0/470068586 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53faffd700 1 -- 172.21.15.115:0/470068586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec005040 con 0x7f53fc0f6a10 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53faffd700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f53ec013510 tx=0x7f53ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec009320 con 0x7f53fc0f6a10 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53ec013f00 con 0x7f53fc0f6a10 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc105cc0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec014d90 con 0x7f53fc0f6a10 2024-01-21T07:36:37.658 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.655+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc106150 con 0x7f53fc0f6a10 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.656+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53ec0263a0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.656+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc045ad0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.656+0000 7f53e37fe700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.656+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53ec0413c0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.656+0000 7f53fa7fc700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:37.659 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.657+0000 7f53fa7fc700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:37.662 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.661+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53ec0126f0 con 0x7f53fc0f6a10 2024-01-21T07:36:37.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.870+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53fc107000 con 0x7f53fc0f6a10 2024-01-21T07:36:37.871 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.870+0000 7f53e37fe700 1 -- 172.21.15.115:0/470068586 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53ec012330 con 0x7f53fc0f6a10 2024-01-21T07:36:37.872 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:37.872 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:37.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 msgr2=0x7f53e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:37.873 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).stop 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 msgr2=0x7f53fc1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f53ec013510 tx=0x7f53ec00b040).stop 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 shutdown_connections 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 --2- 172.21.15.115:0/470068586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53fc0f6a10 0x7f53fc1055d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 >> 172.21.15.115:0/470068586 conn(0x7f53fc0f02e0 msgr2=0x7f53fc0f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 shutdown_connections 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:37.872+0000 7f540134c700 1 -- 172.21.15.115:0/470068586 wait complete. 2024-01-21T07:36:37.874 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:37.870819+0000 mon.smithi107 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.115:0/470068586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:39.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:39.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:37.881196+0000 mgr.smithi107.adejht (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:40.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.373+0000 7f42ee497700 1 -- 172.21.15.115:0/1432939132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 msgr2=0x7f42e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:40.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.373+0000 7f42ee497700 1 --2- 172.21.15.115:0/1432939132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e80f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f42d8009fb0 tx=0x7f42d8009b30).stop 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 -- 172.21.15.115:0/1432939132 shutdown_connections 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 --2- 172.21.15.115:0/1432939132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e80f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 -- 172.21.15.115:0/1432939132 >> 172.21.15.115:0/1432939132 conn(0x7f42e80f0b70 msgr2=0x7f42e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 -- 172.21.15.115:0/1432939132 shutdown_connections 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 -- 172.21.15.115:0/1432939132 wait complete. 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 Processor -- start 2024-01-21T07:36:40.376 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.374+0000 7f42ee497700 1 -- start start 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42ee497700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42ee497700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8105af0 con 0x7f42e80f5410 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48112/0 (socket says 172.21.15.115:48112) 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e7fff700 1 -- 172.21.15.115:0/20445352 learned_addr learned my addr 172.21.15.115:0/20445352 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:40.378 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e7fff700 1 -- 172.21.15.115:0/20445352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8005040 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e7fff700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f42d8013510 tx=0x7f42d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d8009320 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.375+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e8105cf0 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42d8013f00 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e8106180 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d8014d90 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42d80095f0 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e8045ad0 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.376+0000 7f42e4ff9700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 0x7f42d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.377+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42d8040fd0 con 0x7f42e80f5410 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.377+0000 7f42e77fe700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 0x7f42d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:40.379 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.377+0000 7f42e77fe700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 0x7f42d0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f42dc0096a0 tx=0x7f42dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:40.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.381+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42d8014420 con 0x7f42e80f5410 2024-01-21T07:36:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.588+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42e80fae60 con 0x7f42e80f5410 2024-01-21T07:36:40.589 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.588+0000 7f42e4ff9700 1 -- 172.21.15.115:0/20445352 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42d8010230 con 0x7f42e80f5410 2024-01-21T07:36:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:40.590 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 msgr2=0x7f42d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 0x7f42d0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f42dc0096a0 tx=0x7f42dc006b40).stop 2024-01-21T07:36:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 msgr2=0x7f42e8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:40.591 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f42d8013510 tx=0x7f42d800b040).stop 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 shutdown_connections 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f42d002ff60 0x7f42d0032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 --2- 172.21.15.115:0/20445352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42e80f5410 0x7f42e8105600 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 >> 172.21.15.115:0/20445352 conn(0x7f42e80f0b70 msgr2=0x7f42e80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 shutdown_connections 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:40.590+0000 7f42ee497700 1 -- 172.21.15.115:0/20445352 wait complete. 2024-01-21T07:36:40.592 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:41.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:39.881541+0000 mgr.smithi107.adejht (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:41.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:40 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:40.588799+0000 mon.smithi107 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.115:0/20445352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:41.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:41.970 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:43.120 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- 172.21.15.115:0/2904928968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 msgr2=0x7f2ce80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 --2- 172.21.15.115:0/2904928968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce80f55c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f2cd8009fb0 tx=0x7f2cd8009b30).stop 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- 172.21.15.115:0/2904928968 shutdown_connections 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 --2- 172.21.15.115:0/2904928968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce80f55c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- 172.21.15.115:0/2904928968 >> 172.21.15.115:0/2904928968 conn(0x7f2ce80f09a0 msgr2=0x7f2ce80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- 172.21.15.115:0/2904928968 shutdown_connections 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- 172.21.15.115:0/2904928968 wait complete. 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 Processor -- start 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.118+0000 7f2ce759e700 1 -- start start 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce759e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce81058a0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48120/0 (socket says 172.21.15.115:48120) 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce659c700 1 -- 172.21.15.115:0/3189540512 learned_addr learned my addr 172.21.15.115:0/3189540512 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce659c700 1 -- 172.21.15.115:0/3189540512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8005040 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2ce659c700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f2cd8013510 tx=0x7f2cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.119+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd8009320 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce8105aa0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cd8013f00 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd8014dc0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce8105f30 con 0x7f2ce80f51a0 2024-01-21T07:36:43.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce8045ad0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cd80095f0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.120+0000 7f2cd6ffd700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 0x7f2cd0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:43.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.121+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cd80410a0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.121+0000 7f2ce5d9b700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 0x7f2cd0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:43.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.121+0000 7f2ce5d9b700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 0x7f2cd0032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2cdc0096a0 tx=0x7f2cdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:43.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.125+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cd806d2b0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:41.881791+0000 mgr.smithi107.adejht (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:43.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.330+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ce8106de0 con 0x7f2ce80f51a0 2024-01-21T07:36:43.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.331+0000 7f2cd6ffd700 1 -- 172.21.15.115:0/3189540512 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2cd8014420 con 0x7f2ce80f51a0 2024-01-21T07:36:43.333 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:43.333 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 msgr2=0x7f2cd0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 0x7f2cd0032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2cdc0096a0 tx=0x7f2cdc006b40).stop 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 msgr2=0x7f2ce81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f2cd8013510 tx=0x7f2cd800b040).stop 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 shutdown_connections 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f2cd002ffb0 0x7f2cd0032470 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 --2- 172.21.15.115:0/3189540512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ce80f51a0 0x7f2ce81053b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 >> 172.21.15.115:0/3189540512 conn(0x7f2ce80f09a0 msgr2=0x7f2ce80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 shutdown_connections 2024-01-21T07:36:43.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:43.333+0000 7f2ce759e700 1 -- 172.21.15.115:0/3189540512 wait complete. 2024-01-21T07:36:43.336 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:44.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:43.332511+0000 mon.smithi107 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.115:0/3189540512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:44.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:44.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:45.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:43.882207+0000 mgr.smithi107.adejht (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 -- 172.21.15.115:0/2473083554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 msgr2=0x7f95d80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/2473083554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f90f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f95cc009fb0 tx=0x7f95cc009b30).stop 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 -- 172.21.15.115:0/2473083554 shutdown_connections 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/2473083554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f90f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 -- 172.21.15.115:0/2473083554 >> 172.21.15.115:0/2473083554 conn(0x7f95d80f0b70 msgr2=0x7f95d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 -- 172.21.15.115:0/2473083554 shutdown_connections 2024-01-21T07:36:45.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.883+0000 7f95dd9ed700 1 -- 172.21.15.115:0/2473083554 wait complete. 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95dd9ed700 1 Processor -- start 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95dd9ed700 1 -- start start 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95dd9ed700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95dd9ed700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80f4ff0 con 0x7f95d8060280 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95d6ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:45.888 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95d6ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48144/0 (socket says 172.21.15.115:48144) 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.884+0000 7f95d6ffd700 1 -- 172.21.15.115:0/695246356 learned_addr learned my addr 172.21.15.115:0/695246356 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95d6ffd700 1 -- 172.21.15.115:0/695246356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc005040 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95d6ffd700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f95cc013510 tx=0x7f95cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95cc009320 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d80f5190 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95cc013f00 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95cc014dc0 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.885+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d80f5620 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95cc0095f0 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d8059100 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95bf7fe700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 0x7f95c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95cc041000 con 0x7f95d8060280 2024-01-21T07:36:45.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95d67fc700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 0x7f95c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:45.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.886+0000 7f95d67fc700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 0x7f95c0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f95c80096a0 tx=0x7f95c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:45.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:45.890+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95cc014420 con 0x7f95d8060280 2024-01-21T07:36:46.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.100+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95d80fbf50 con 0x7f95d8060280 2024-01-21T07:36:46.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.101+0000 7f95bf7fe700 1 -- 172.21.15.115:0/695246356 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95cc010230 con 0x7f95d8060280 2024-01-21T07:36:46.104 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:46.104 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 msgr2=0x7f95c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 0x7f95c0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f95c80096a0 tx=0x7f95c8006b40).stop 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 msgr2=0x7f95d80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f95cc013510 tx=0x7f95cc00b040).stop 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 shutdown_connections 2024-01-21T07:36:46.105 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f95c002ff60 0x7f95c0032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:46.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 --2- 172.21.15.115:0/695246356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8060280 0x7f95d80f6b70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:46.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 >> 172.21.15.115:0/695246356 conn(0x7f95d80f0b70 msgr2=0x7f95d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:46.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 shutdown_connections 2024-01-21T07:36:46.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:46.103+0000 7f95dd9ed700 1 -- 172.21.15.115:0/695246356 wait complete. 2024-01-21T07:36:46.106 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:45.882537+0000 mgr.smithi107.adejht (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:46.102498+0000 mon.smithi107 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.115:0/695246356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:47.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:47.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:47.882887+0000 mgr.smithi107.adejht (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 -- 172.21.15.115:0/930985804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 msgr2=0x7f00fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 --2- 172.21.15.115:0/930985804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc0f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f00ec009fb0 tx=0x7f00ec009b30).stop 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 -- 172.21.15.115:0/930985804 shutdown_connections 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 --2- 172.21.15.115:0/930985804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc0f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 -- 172.21.15.115:0/930985804 >> 172.21.15.115:0/930985804 conn(0x7f00fc0f0b70 msgr2=0x7f00fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 -- 172.21.15.115:0/930985804 shutdown_connections 2024-01-21T07:36:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.614+0000 7f0100c52700 1 -- 172.21.15.115:0/930985804 wait complete. 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f0100c52700 1 Processor -- start 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f0100c52700 1 -- start start 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f0100c52700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f0100c52700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc060240 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f00fa59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f00fa59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48158/0 (socket says 172.21.15.115:48158) 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f00fa59c700 1 -- 172.21.15.115:0/4035575781 learned_addr learned my addr 172.21.15.115:0/4035575781 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.615+0000 7f00fa59c700 1 -- 172.21.15.115:0/4035575781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f00fa59c700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f00ec013510 tx=0x7f00ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ec009320 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc060440 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00ec013f00 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ec014d90 con 0x7f00fc0f5410 2024-01-21T07:36:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc109ae0 con 0x7f00fc0f5410 2024-01-21T07:36:48.620 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00ec0095f0 con 0x7f00fc0f5410 2024-01-21T07:36:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.616+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc045ad0 con 0x7f00fc0f5410 2024-01-21T07:36:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.617+0000 7f00eaffd700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 0x7f00e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.617+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00ec0410a0 con 0x7f00fc0f5410 2024-01-21T07:36:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.617+0000 7f00f9d9b700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 0x7f00e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.617+0000 7f00f9d9b700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 0x7f00e4032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f00f00096a0 tx=0x7f00f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:48.623 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.621+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00ec01d7f0 con 0x7f00fc0f5410 2024-01-21T07:36:48.830 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.828+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00fc0fae60 con 0x7f00fc0f5410 2024-01-21T07:36:48.831 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.828+0000 7f00eaffd700 1 -- 172.21.15.115:0/4035575781 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f00ec010230 con 0x7f00fc0f5410 2024-01-21T07:36:48.831 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:48.831 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:48.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 msgr2=0x7f00e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:48.832 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 0x7f00e4032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f00f00096a0 tx=0x7f00f0006b40).stop 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 msgr2=0x7f00fc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f00ec013510 tx=0x7f00ec00b040).stop 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 shutdown_connections 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f00e402ff60 0x7f00e4032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 --2- 172.21.15.115:0/4035575781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00fc0f5410 0x7f00fc109430 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 >> 172.21.15.115:0/4035575781 conn(0x7f00fc0f0b70 msgr2=0x7f00fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 shutdown_connections 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:48.830+0000 7f0100c52700 1 -- 172.21.15.115:0/4035575781 wait complete. 2024-01-21T07:36:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:49.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:48 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:48.829927+0000 mon.smithi107 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.115:0/4035575781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:50.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:50.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:50.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:49.883225+0000 mgr.smithi107.adejht (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:36:50.354109+0000 mgr.smithi107.adejht (mgr.14162) 237 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi107 ... 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Verifying port 9100 ... 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 1 from systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: systemctl: stderr Job for ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107.service failed because a timeout was exceeded. 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: systemctl: stderr See "systemctl status ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107.service" and "journalctl -xe" for details. 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T07:36:51.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: main() 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: r = ctx.func(ctx) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: return func(ctx) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ports=daemon_ports) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T07:36:51.317 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: RuntimeError: Failed command: systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: yield (conn, connr) 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: code, '\n'.join(err))) 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi107 ... 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Verifying port 9100 ... 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Non-zero exit code 1 from systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: systemctl: stderr Job for ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107.service failed because a timeout was exceeded. 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: systemctl: stderr See "systemctl status ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107.service" and "journalctl -xe" for details. 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: Traceback (most recent call last): 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: main() 2024-01-21T07:36:51.318 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: r = ctx.func(ctx) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: return func(ctx) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: ports=daemon_ports) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: File "/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: RuntimeError: Failed command: systemctl start ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@node-exporter.smithi107 2024-01-21T07:36:51.319 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cephadm 2024-01-21T07:36:50.359816+0000 mgr.smithi107.adejht (mgr.14162) 238 : cephadm [INF] Deploying daemon prometheus.smithi107 on smithi107 2024-01-21T07:36:51.369 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- 172.21.15.115:0/1690411941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 msgr2=0x7ff6600f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:51.369 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 --2- 172.21.15.115:0/1690411941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f8f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7ff650009fb0 tx=0x7ff650009b30).stop 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- 172.21.15.115:0/1690411941 shutdown_connections 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 --2- 172.21.15.115:0/1690411941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f8f20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- 172.21.15.115:0/1690411941 >> 172.21.15.115:0/1690411941 conn(0x7ff6600f09a0 msgr2=0x7ff6600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- 172.21.15.115:0/1690411941 shutdown_connections 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- 172.21.15.115:0/1690411941 wait complete. 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 Processor -- start 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- start start 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.366+0000 7ff66614c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6600f4e20 con 0x7ff6600601e0 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66514a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66514a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:42248/0 (socket says 172.21.15.115:42248) 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66514a700 1 -- 172.21.15.115:0/3743411865 learned_addr learned my addr 172.21.15.115:0/3743411865 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:51.370 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66514a700 1 -- 172.21.15.115:0/3743411865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff650005040 con 0x7ff6600601e0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66514a700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff650013510 tx=0x7ff65000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff650009320 con 0x7ff6600601e0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff650013f00 con 0x7ff6600601e0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6600f4fc0 con 0x7ff6600601e0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.367+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff650014d90 con 0x7ff6600601e0 2024-01-21T07:36:51.371 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.368+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6600f5450 con 0x7ff6600601e0 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.368+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6500095f0 con 0x7ff6600601e0 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.368+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff660059100 con 0x7ff6600601e0 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.368+0000 7ff655ffb700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.368+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff650041080 con 0x7ff6600601e0 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.369+0000 7ff664949700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:51.372 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.369+0000 7ff664949700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 0x7ff64c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff65c0096a0 tx=0x7ff65c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:51.375 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.373+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff650014420 con 0x7ff6600601e0 2024-01-21T07:36:51.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.580+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6600fbcd0 con 0x7ff6600601e0 2024-01-21T07:36:51.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.580+0000 7ff655ffb700 1 -- 172.21.15.115:0/3743411865 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff650010230 con 0x7ff6600601e0 2024-01-21T07:36:51.583 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 msgr2=0x7ff64c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 0x7ff64c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff65c0096a0 tx=0x7ff65c006b40).stop 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 msgr2=0x7ff6600f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff650013510 tx=0x7ff65000b040).stop 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 shutdown_connections 2024-01-21T07:36:51.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff64c02ff60 0x7ff64c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:51.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 --2- 172.21.15.115:0/3743411865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6600601e0 0x7ff6600f69a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:51.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 >> 172.21.15.115:0/3743411865 conn(0x7ff6600f09a0 msgr2=0x7ff6600f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:51.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 shutdown_connections 2024-01-21T07:36:51.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:51.582+0000 7ff66614c700 1 -- 172.21.15.115:0/3743411865 wait complete. 2024-01-21T07:36:51.585 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:52.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:51.581740+0000 mon.smithi107 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.115:0/3743411865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:52.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:51 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:51.883620+0000 mgr.smithi107.adejht (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:52.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:52.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:54.152 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/1504858279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 msgr2=0x7fc4b40f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:54.152 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/1504858279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b40f5df0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fc4a8009fb0 tx=0x7fc4a8009b30).stop 2024-01-21T07:36:54.152 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/1504858279 shutdown_connections 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/1504858279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b40f5df0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/1504858279 >> 172.21.15.115:0/1504858279 conn(0x7fc4b40f02e0 msgr2=0x7fc4b40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/1504858279 shutdown_connections 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.147+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/1504858279 wait complete. 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b8ca7700 1 Processor -- start 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b8ca7700 1 -- start start 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b8ca7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b8ca7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4105ab0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:42262/0 (socket says 172.21.15.115:42262) 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b259c700 1 -- 172.21.15.115:0/4259816413 learned_addr learned my addr 172.21.15.115:0/4259816413 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b259c700 1 -- 172.21.15.115:0/4259816413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8005040 con 0x7fc4b40f59d0 2024-01-21T07:36:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.148+0000 7fc4b259c700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc4a8013510 tx=0x7fc4a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a8009320 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4a8013f00 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4105cb0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a8014d90 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b4106140 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.149+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4a80095f0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.150+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4045ad0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.150+0000 7fc4a2ffd700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 0x7fc49c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.150+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4a8040fc0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.150+0000 7fc4b1d9b700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 0x7fc49c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:54.154 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.150+0000 7fc4b1d9b700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 0x7fc49c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc4a40096a0 tx=0x7fc4a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:54.156 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.154+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4a8014420 con 0x7fc4b40f59d0 2024-01-21T07:36:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:53.884014+0000 mgr.smithi107.adejht (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:54.362 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.360+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4b4106ff0 con 0x7fc4b40f59d0 2024-01-21T07:36:54.363 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.361+0000 7fc4a2ffd700 1 -- 172.21.15.115:0/4259816413 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc4a8010230 con 0x7fc4b40f59d0 2024-01-21T07:36:54.363 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:54.363 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.362+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 msgr2=0x7fc49c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.362+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 0x7fc49c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc4a40096a0 tx=0x7fc4a4006b40).stop 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.362+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 msgr2=0x7fc4b41055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.362+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc4a8013510 tx=0x7fc4a800b040).stop 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 shutdown_connections 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fc49c02ff60 0x7fc49c032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 --2- 172.21.15.115:0/4259816413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4b40f59d0 0x7fc4b41055c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 >> 172.21.15.115:0/4259816413 conn(0x7fc4b40f02e0 msgr2=0x7fc4b40f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 shutdown_connections 2024-01-21T07:36:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:54.363+0000 7fc4b8ca7700 1 -- 172.21.15.115:0/4259816413 wait complete. 2024-01-21T07:36:54.366 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:55.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:54 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:54.362158+0000 mon.smithi107 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.115:0/4259816413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:55.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:55.743 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:56.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:55.884452+0000 mgr.smithi107.adejht (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:56.862 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 -- 172.21.15.115:0/1489286823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 msgr2=0x7f58ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:56.862 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 --2- 172.21.15.115:0/1489286823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec0f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f58e0009fb0 tx=0x7f58e0009b30).stop 2024-01-21T07:36:56.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 -- 172.21.15.115:0/1489286823 shutdown_connections 2024-01-21T07:36:56.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 --2- 172.21.15.115:0/1489286823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec0f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:56.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 -- 172.21.15.115:0/1489286823 >> 172.21.15.115:0/1489286823 conn(0x7f58ec0f0b70 msgr2=0x7f58ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:56.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 -- 172.21.15.115:0/1489286823 shutdown_connections 2024-01-21T07:36:56.863 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.860+0000 7f58f159a700 1 -- 172.21.15.115:0/1489286823 wait complete. 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58f159a700 1 Processor -- start 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58f159a700 1 -- start start 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58f159a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58f159a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec106c20 con 0x7f58ec0f5410 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58eaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58eaffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:42286/0 (socket says 172.21.15.115:42286) 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.861+0000 7f58eaffd700 1 -- 172.21.15.115:0/3596359897 learned_addr learned my addr 172.21.15.115:0/3596359897 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:56.865 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58eaffd700 1 -- 172.21.15.115:0/3596359897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e0005040 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58eaffd700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f58e0013510 tx=0x7f58e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e0009320 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec106e20 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58e0013f00 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e0014d90 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.862+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec1072b0 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58e0014420 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec045ad0 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58d37fe700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 0x7f58d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58e00410c0 con 0x7f58ec0f5410 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58ea7fc700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 0x7f58d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:56.866 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.863+0000 7f58ea7fc700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 0x7f58d4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f58dc0096a0 tx=0x7f58dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:56.870 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:56.868+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58e00095f0 con 0x7f58ec0f5410 2024-01-21T07:36:57.076 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.074+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58ec0fae60 con 0x7f58ec0f5410 2024-01-21T07:36:57.077 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.075+0000 7f58d37fe700 1 -- 172.21.15.115:0/3596359897 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f58e001db70 con 0x7f58ec0f5410 2024-01-21T07:36:57.077 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:57.077 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:57.078 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 msgr2=0x7f58d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:57.078 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 0x7f58d4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f58dc0096a0 tx=0x7f58dc006b40).stop 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 msgr2=0x7f58ec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f58e0013510 tx=0x7f58e000b040).stop 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 shutdown_connections 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f58d402ff60 0x7f58d4032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 --2- 172.21.15.115:0/3596359897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f58ec0f5410 0x7f58ec106730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 >> 172.21.15.115:0/3596359897 conn(0x7f58ec0f0b70 msgr2=0x7f58ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 shutdown_connections 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:57.076+0000 7f58f159a700 1 -- 172.21.15.115:0/3596359897 wait complete. 2024-01-21T07:36:57.079 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:36:58.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:57.076165+0000 mon.smithi107 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.115:0/3596359897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:36:58.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:36:58.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:36:59.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:57.884878+0000 mgr.smithi107.adejht (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:36:59.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 -- 172.21.15.115:0/3181812761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 msgr2=0x7f30b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:59.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 --2- 172.21.15.115:0/3181812761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b40f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f30ac009fb0 tx=0x7f30ac009b30).stop 2024-01-21T07:36:59.576 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 -- 172.21.15.115:0/3181812761 shutdown_connections 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 --2- 172.21.15.115:0/3181812761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b40f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 -- 172.21.15.115:0/3181812761 >> 172.21.15.115:0/3181812761 conn(0x7f30b40f1c70 msgr2=0x7f30b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 -- 172.21.15.115:0/3181812761 shutdown_connections 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.573+0000 7f30bbc07700 1 -- 172.21.15.115:0/3181812761 wait complete. 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30bbc07700 1 Processor -- start 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30bbc07700 1 -- start start 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30bbc07700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30bbc07700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4069e20 con 0x7f30b40f6510 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30b99a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30b99a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:56016/0 (socket says 172.21.15.115:56016) 2024-01-21T07:36:59.577 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.574+0000 7f30b99a3700 1 -- 172.21.15.115:0/1394249644 learned_addr learned my addr 172.21.15.115:0/1394249644 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30b99a3700 1 -- 172.21.15.115:0/1394249644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac005040 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30b99a3700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f30ac013510 tx=0x7f30ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac009320 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b4069fc0 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30ac013f00 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac014d90 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.575+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b406a450 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30ac0095f0 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b4059100 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30b27fc700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 0x7f309c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30ac041080 con 0x7f30b40f6510 2024-01-21T07:36:59.579 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30b91a2700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 0x7f309c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:36:59.580 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.576+0000 7f30b91a2700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 0x7f309c032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f30a80096a0 tx=0x7f30a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:36:59.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.581+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30ac014420 con 0x7f30b40f6510 2024-01-21T07:36:59.787 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.785+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30b40fbf60 con 0x7f30b40f6510 2024-01-21T07:36:59.788 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.786+0000 7f30b27fc700 1 -- 172.21.15.115:0/1394249644 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30ac01d7f0 con 0x7f30b40f6510 2024-01-21T07:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:36:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 msgr2=0x7f309c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 0x7f309c032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f30a80096a0 tx=0x7f30a8006b40).stop 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 msgr2=0x7f30b406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f30ac013510 tx=0x7f30ac00b040).stop 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 shutdown_connections 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f309c02ff60 0x7f309c032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 --2- 172.21.15.115:0/1394249644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30b40f6510 0x7f30b406b9a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 >> 172.21.15.115:0/1394249644 conn(0x7f30b40f1c70 msgr2=0x7f30b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 shutdown_connections 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:36:59.787+0000 7f30bbc07700 1 -- 172.21.15.115:0/1394249644 wait complete. 2024-01-21T07:36:59.790 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:00.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:36:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:36:59.787161+0000 mon.smithi107 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.115:0/1394249644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:01.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:01.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:01.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:36:59.885255+0000 mgr.smithi107.adejht (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 -- 172.21.15.115:0/3141588567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 msgr2=0x7f5f540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3141588567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f540f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f5f44009fb0 tx=0x7f5f44009b30).stop 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 -- 172.21.15.115:0/3141588567 shutdown_connections 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3141588567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f540f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 -- 172.21.15.115:0/3141588567 >> 172.21.15.115:0/3141588567 conn(0x7f5f540f09a0 msgr2=0x7f5f540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 -- 172.21.15.115:0/3141588567 shutdown_connections 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 -- 172.21.15.115:0/3141588567 wait complete. 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.298+0000 7f5f5882b700 1 Processor -- start 2024-01-21T07:37:02.302 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f5882b700 1 -- start start 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f5882b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f5882b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54060240 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f537fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f537fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:56036/0 (socket says 172.21.15.115:56036) 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f537fe700 1 -- 172.21.15.115:0/3996589667 learned_addr learned my addr 172.21.15.115:0/3996589667 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f537fe700 1 -- 172.21.15.115:0/3996589667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f44005040 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.299+0000 7f5f537fe700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f5f44009830 tx=0x7f5f4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f44009320 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f440136c0 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f54060440 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f4401ee90 con 0x7f5f540f51a0 2024-01-21T07:37:02.303 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f54109860 con 0x7f5f540f51a0 2024-01-21T07:37:02.304 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.300+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f4401e420 con 0x7f5f540f51a0 2024-01-21T07:37:02.304 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.301+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f54045ad0 con 0x7f5f540f51a0 2024-01-21T07:37:02.304 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.301+0000 7f5f3bfff700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 0x7f5f3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:02.304 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.301+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f44041120 con 0x7f5f540f51a0 2024-01-21T07:37:02.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.301+0000 7f5f52ffd700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 0x7f5f3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:02.305 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.301+0000 7f5f52ffd700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 0x7f5f3c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5f480096a0 tx=0x7f5f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:02.308 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.306+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f440095f0 con 0x7f5f540f51a0 2024-01-21T07:37:02.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.512+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f5410a080 con 0x7f5f540f51a0 2024-01-21T07:37:02.515 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.513+0000 7f5f3bfff700 1 -- 172.21.15.115:0/3996589667 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f4401cb10 con 0x7f5f540f51a0 2024-01-21T07:37:02.515 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:02.515 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:02.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 msgr2=0x7f5f3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:02.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 0x7f5f3c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5f480096a0 tx=0x7f5f48006b40).stop 2024-01-21T07:37:02.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 msgr2=0x7f5f541091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:02.517 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f5f44009830 tx=0x7f5f4400b040).stop 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 shutdown_connections 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f5f3c02ff10 0x7f5f3c0323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 --2- 172.21.15.115:0/3996589667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f540f51a0 0x7f5f541091b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 >> 172.21.15.115:0/3996589667 conn(0x7f5f540f09a0 msgr2=0x7f5f540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 shutdown_connections 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:02.515+0000 7f5f5882b700 1 -- 172.21.15.115:0/3996589667 wait complete. 2024-01-21T07:37:02.518 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:03.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:01.885625+0000 mgr.smithi107.adejht (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:03.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:02.514630+0000 mon.smithi107 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.115:0/3996589667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:03.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:03.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:03.886037+0000 mgr.smithi107.adejht (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:05.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.120+0000 7f8983b0e700 1 -- 172.21.15.115:0/2167301500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 msgr2=0x7f897c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:05.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.120+0000 7f8983b0e700 1 --2- 172.21.15.115:0/2167301500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c0f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f896c009fb0 tx=0x7f896c009b30).stop 2024-01-21T07:37:05.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.120+0000 7f8983b0e700 1 -- 172.21.15.115:0/2167301500 shutdown_connections 2024-01-21T07:37:05.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.120+0000 7f8983b0e700 1 --2- 172.21.15.115:0/2167301500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c0f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:05.123 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.120+0000 7f8983b0e700 1 -- 172.21.15.115:0/2167301500 >> 172.21.15.115:0/2167301500 conn(0x7f897c0f0b70 msgr2=0x7f897c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:05.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 -- 172.21.15.115:0/2167301500 shutdown_connections 2024-01-21T07:37:05.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 -- 172.21.15.115:0/2167301500 wait complete. 2024-01-21T07:37:05.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 Processor -- start 2024-01-21T07:37:05.124 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 -- start start 2024-01-21T07:37:05.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:05.125 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.121+0000 7f8983b0e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c105b20 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89818aa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89818aa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:56054/0 (socket says 172.21.15.115:56054) 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89818aa700 1 -- 172.21.15.115:0/4009779274 learned_addr learned my addr 172.21.15.115:0/4009779274 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89818aa700 1 -- 172.21.15.115:0/4009779274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c005040 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89818aa700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f896c013510 tx=0x7f896c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c009320 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897c105d20 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f896c013f00 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f896c014d90 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.122+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897c1061b0 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.123+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897c045ad0 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.123+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f896c0095f0 con 0x7f897c0f5410 2024-01-21T07:37:05.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.123+0000 7f89727fc700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 0x7f8968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:05.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.123+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f896c041000 con 0x7f897c0f5410 2024-01-21T07:37:05.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.124+0000 7f89810a9700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 0x7f8968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:05.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.124+0000 7f89810a9700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 0x7f8968032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f89780096a0 tx=0x7f8978006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:05.130 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.128+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f896c01d7f0 con 0x7f897c0f5410 2024-01-21T07:37:05.333 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.331+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f897c0fae60 con 0x7f897c0f5410 2024-01-21T07:37:05.334 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.332+0000 7f89727fc700 1 -- 172.21.15.115:0/4009779274 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f896c010230 con 0x7f897c0f5410 2024-01-21T07:37:05.334 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:05.334 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:05.335 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 msgr2=0x7f8968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 0x7f8968032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f89780096a0 tx=0x7f8978006b40).stop 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 msgr2=0x7f897c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f896c013510 tx=0x7f896c00b040).stop 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 shutdown_connections 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f896802ff60 0x7f8968032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 --2- 172.21.15.115:0/4009779274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f897c0f5410 0x7f897c105630 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 >> 172.21.15.115:0/4009779274 conn(0x7f897c0f0b70 msgr2=0x7f897c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 shutdown_connections 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:05.333+0000 7f8983b0e700 1 -- 172.21.15.115:0/4009779274 wait complete. 2024-01-21T07:37:05.336 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:05.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:05.333110+0000 mon.smithi107 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.115:0/4009779274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:06.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:06.729 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:06.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:05.886431+0000 mgr.smithi107.adejht (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 -- 172.21.15.115:0/3207322775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 msgr2=0x7fd7940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 --2- 172.21.15.115:0/3207322775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd7940f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fd77c009fb0 tx=0x7fd77c009b30).stop 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 -- 172.21.15.115:0/3207322775 shutdown_connections 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 --2- 172.21.15.115:0/3207322775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd7940f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 -- 172.21.15.115:0/3207322775 >> 172.21.15.115:0/3207322775 conn(0x7fd7940f0b70 msgr2=0x7fd7940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 -- 172.21.15.115:0/3207322775 shutdown_connections 2024-01-21T07:37:07.850 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.846+0000 7fd798baa700 1 -- 172.21.15.115:0/3207322775 wait complete. 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd798baa700 1 Processor -- start 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd798baa700 1 -- start start 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd798baa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd798baa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd794060240 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd79259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd79259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:56066/0 (socket says 172.21.15.115:56066) 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.847+0000 7fd79259c700 1 -- 172.21.15.115:0/3217157727 learned_addr learned my addr 172.21.15.115:0/3217157727 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd79259c700 1 -- 172.21.15.115:0/3217157727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c005040 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd79259c700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd77c013510 tx=0x7fd77c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd77c009320 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794060440 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd77c013f00 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd77c014d90 con 0x7fd7940f5410 2024-01-21T07:37:07.851 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.848+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd794109ae0 con 0x7fd7940f5410 2024-01-21T07:37:07.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd77c0263a0 con 0x7fd7940f5410 2024-01-21T07:37:07.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd78affd700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 0x7fd780032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:07.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd77c0410a0 con 0x7fd7940f5410 2024-01-21T07:37:07.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd794045ad0 con 0x7fd7940f5410 2024-01-21T07:37:07.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd791d9b700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 0x7fd780032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:07.853 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.849+0000 7fd791d9b700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 0x7fd780032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd7840096a0 tx=0x7fd784006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:07.856 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:07.854+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd77c012920 con 0x7fd7940f5410 2024-01-21T07:37:08.063 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.061+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd79410a1f0 con 0x7fd7940f5410 2024-01-21T07:37:08.063 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.062+0000 7fd78affd700 1 -- 172.21.15.115:0/3217157727 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd77c02b020 con 0x7fd7940f5410 2024-01-21T07:37:08.064 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:08.064 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:08.065 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 msgr2=0x7fd780032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:08.065 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 0x7fd780032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd7840096a0 tx=0x7fd784006b40).stop 2024-01-21T07:37:08.065 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 msgr2=0x7fd794109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:08.065 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd77c013510 tx=0x7fd77c00b040).stop 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 shutdown_connections 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fd78002ff60 0x7fd780032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 --2- 172.21.15.115:0/3217157727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7940f5410 0x7fd794109430 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 >> 172.21.15.115:0/3217157727 conn(0x7fd7940f0b70 msgr2=0x7fd7940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 shutdown_connections 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:08.063+0000 7fd798baa700 1 -- 172.21.15.115:0/3217157727 wait complete. 2024-01-21T07:37:08.066 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:08.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:07 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:07.886690+0000 mgr.smithi107.adejht (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:09.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:08 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:08.063058+0000 mon.smithi107 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.115:0/3217157727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:09.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:09.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:09.887024+0000 mgr.smithi107.adejht (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:09.953176+0000 mon.smithi107 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/mirror_snapshot_schedule"}]: dispatch 2024-01-21T07:37:10.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:09 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:09.953386+0000 mon.smithi107 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.107:0/3326511127' entity='mgr.smithi107.adejht' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi107.adejht/trash_purge_schedule"}]: dispatch 2024-01-21T07:37:10.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 -- 172.21.15.115:0/3509514752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 msgr2=0x7f10500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:10.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 --2- 172.21.15.115:0/3509514752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f1040009fb0 tx=0x7f1040009b30).stop 2024-01-21T07:37:10.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 -- 172.21.15.115:0/3509514752 shutdown_connections 2024-01-21T07:37:10.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 --2- 172.21.15.115:0/3509514752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:10.638 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 -- 172.21.15.115:0/3509514752 >> 172.21.15.115:0/3509514752 conn(0x7f10500f0b70 msgr2=0x7f10500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:10.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 -- 172.21.15.115:0/3509514752 shutdown_connections 2024-01-21T07:37:10.639 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.636+0000 7f105563a700 1 -- 172.21.15.115:0/3509514752 wait complete. 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f105563a700 1 Processor -- start 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f105563a700 1 -- start start 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f105563a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f105563a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10500fb820 con 0x7f10500f5410 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f104effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f104effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:37324/0 (socket says 172.21.15.115:37324) 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f104effd700 1 -- 172.21.15.115:0/4103327680 learned_addr learned my addr 172.21.15.115:0/4103327680 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f104effd700 1 -- 172.21.15.115:0/4103327680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1040005040 con 0x7f10500f5410 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.637+0000 7f104effd700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f1040013510 tx=0x7f104000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1040009320 con 0x7f10500f5410 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1040013f00 con 0x7f10500f5410 2024-01-21T07:37:10.641 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10500fb9c0 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1040014d90 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10500fbe50 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.638+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1040014420 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.639+0000 7f10377fe700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 0x7f1038032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.639+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1050059100 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.639+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1040041020 con 0x7f10500f5410 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.639+0000 7f104e7fc700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 0x7f1038032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.639+0000 7f104e7fc700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 0x7f1038032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f10440096a0 tx=0x7f1044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:10.645 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.643+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10400095f0 con 0x7f10500f5410 2024-01-21T07:37:10.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.850+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1050045ad0 con 0x7f10500f5410 2024-01-21T07:37:10.852 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.850+0000 7f10377fe700 1 -- 172.21.15.115:0/4103327680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f104001db70 con 0x7f10500f5410 2024-01-21T07:37:10.852 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:10.853 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 msgr2=0x7f1038032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 0x7f1038032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f10440096a0 tx=0x7f1044006b40).stop 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 msgr2=0x7f10500fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f1040013510 tx=0x7f104000b040).stop 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 shutdown_connections 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f103802ff60 0x7f1038032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 --2- 172.21.15.115:0/4103327680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10500f5410 0x7f10500fd3a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 >> 172.21.15.115:0/4103327680 conn(0x7f10500f0b70 msgr2=0x7f10500f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 shutdown_connections 2024-01-21T07:37:10.855 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:10.852+0000 7f105563a700 1 -- 172.21.15.115:0/4103327680 wait complete. 2024-01-21T07:37:10.855 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:11.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:10 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:10.851702+0000 mon.smithi107 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.115:0/4103327680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:12.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:12.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:13.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:12 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:11.887379+0000 mgr.smithi107.adejht (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.512+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/1768620333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 msgr2=0x7fdfc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.512+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/1768620333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc00f55c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fdfbc009fb0 tx=0x7fdfbc009b30).stop 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/1768620333 shutdown_connections 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/1768620333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc00f55c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/1768620333 >> 172.21.15.115:0/1768620333 conn(0x7fdfc00f09a0 msgr2=0x7fdfc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/1768620333 shutdown_connections 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/1768620333 wait complete. 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 Processor -- start 2024-01-21T07:37:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- start start 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc6c8f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc0106990 con 0x7fdfc00f51a0 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.513+0000 7fdfc5c8d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc5c8d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:37342/0 (socket says 172.21.15.115:37342) 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc5c8d700 1 -- 172.21.15.115:0/4249177474 learned_addr learned my addr 172.21.15.115:0/4249177474 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc5c8d700 1 -- 172.21.15.115:0/4249177474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc005040 con 0x7fdfc00f51a0 2024-01-21T07:37:13.520 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc5c8d700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fdfbc013510 tx=0x7fdfbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfbc009320 con 0x7fdfc00f51a0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc0106b90 con 0x7fdfc00f51a0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfbc013f00 con 0x7fdfc00f51a0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfbc014dc0 con 0x7fdfc00f51a0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.514+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc0107020 con 0x7fdfc00f51a0 2024-01-21T07:37:13.521 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.515+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfbc01d3d0 con 0x7fdfc00f51a0 2024-01-21T07:37:13.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.515+0000 7fdfb67fc700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 0x7fdfac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:13.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.515+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfbc041030 con 0x7fdfc00f51a0 2024-01-21T07:37:13.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.515+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc0045ad0 con 0x7fdfc00f51a0 2024-01-21T07:37:13.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.515+0000 7fdfc548c700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 0x7fdfac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:13.522 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.516+0000 7fdfc548c700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 0x7fdfac032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:13.523 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.520+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfbc012920 con 0x7fdfc00f51a0 2024-01-21T07:37:13.725 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.723+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfc00facd0 con 0x7fdfc00f51a0 2024-01-21T07:37:13.727 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.724+0000 7fdfb67fc700 1 -- 172.21.15.115:0/4249177474 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfbc02b020 con 0x7fdfc00f51a0 2024-01-21T07:37:13.728 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:13.728 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 msgr2=0x7fdfac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 0x7fdfac032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).stop 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 msgr2=0x7fdfc01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fdfbc013510 tx=0x7fdfbc00b040).stop 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 shutdown_connections 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fdfac02ff60 0x7fdfac032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 --2- 172.21.15.115:0/4249177474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfc00f51a0 0x7fdfc01064a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 >> 172.21.15.115:0/4249177474 conn(0x7fdfc00f09a0 msgr2=0x7fdfc005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 shutdown_connections 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:13.725+0000 7fdfc6c8f700 1 -- 172.21.15.115:0/4249177474 wait complete. 2024-01-21T07:37:13.729 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:14.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:13 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:13.725123+0000 mon.smithi107 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.115:0/4249177474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:15.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:15.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:15.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:14 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:13.887700+0000 mgr.smithi107.adejht (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:16.252 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 -- 172.21.15.115:0/1320435354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 msgr2=0x7f1c980f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/1320435354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c980f4c20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1c8c009fb0 tx=0x7f1c8c009b30).stop 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 -- 172.21.15.115:0/1320435354 shutdown_connections 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/1320435354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c980f4c20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 -- 172.21.15.115:0/1320435354 >> 172.21.15.115:0/1320435354 conn(0x7f1c980f0260 msgr2=0x7f1c980f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 -- 172.21.15.115:0/1320435354 shutdown_connections 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.249+0000 7f1c9d45a700 1 -- 172.21.15.115:0/1320435354 wait complete. 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c9d45a700 1 Processor -- start 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c9d45a700 1 -- start start 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c9d45a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c9d45a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98106ba0 con 0x7f1c980f4800 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c96ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c96ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:37368/0 (socket says 172.21.15.115:37368) 2024-01-21T07:37:16.253 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.250+0000 7f1c96ffd700 1 -- 172.21.15.115:0/588439670 learned_addr learned my addr 172.21.15.115:0/588439670 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c96ffd700 1 -- 172.21.15.115:0/588439670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c005040 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c96ffd700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1c8c009830 tx=0x7f1c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c8c009320 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c98106da0 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c8c0136c0 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c8c01ee90 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.251+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c98107230 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.252+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c8c01e420 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.252+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c98045ad0 con 0x7f1c980f4800 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.252+0000 7f1c7f7fe700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 0x7f1c800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:16.255 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.252+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c8c040f30 con 0x7f1c980f4800 2024-01-21T07:37:16.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.252+0000 7f1c967fc700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 0x7f1c800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:16.256 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.253+0000 7f1c967fc700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 0x7f1c800323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1c880096a0 tx=0x7f1c88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:16.259 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.257+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c8c01cb80 con 0x7f1c980f4800 2024-01-21T07:37:16.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:15 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:15.888040+0000 mgr.smithi107.adejht (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:16.467 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.464+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c981080e0 con 0x7f1c980f4800 2024-01-21T07:37:16.467 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.465+0000 7f1c7f7fe700 1 -- 172.21.15.115:0/588439670 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c8c02c2d0 con 0x7f1c980f4800 2024-01-21T07:37:16.467 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:16.468 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 msgr2=0x7f1c800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 0x7f1c800323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1c880096a0 tx=0x7f1c88006b40).stop 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 msgr2=0x7f1c981066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1c8c009830 tx=0x7f1c8c00b040).stop 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 shutdown_connections 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f1c8002ff10 0x7f1c800323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 --2- 172.21.15.115:0/588439670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c980f4800 0x7f1c981066b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 >> 172.21.15.115:0/588439670 conn(0x7f1c980f0260 msgr2=0x7f1c980f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:16.470 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 shutdown_connections 2024-01-21T07:37:16.470 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:16.467+0000 7f1c9d45a700 1 -- 172.21.15.115:0/588439670 wait complete. 2024-01-21T07:37:16.470 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:17.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:16 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:16.466644+0000 mon.smithi107 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.115:0/588439670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:17.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:17.858 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:18.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:17 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:17.888441+0000 mgr.smithi107.adejht (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 -- 172.21.15.115:0/1515974622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 msgr2=0x7fcfa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 --2- 172.21.15.115:0/1515974622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa40f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fcf94009fb0 tx=0x7fcf94009b30).stop 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 -- 172.21.15.115:0/1515974622 shutdown_connections 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 --2- 172.21.15.115:0/1515974622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa40f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 -- 172.21.15.115:0/1515974622 >> 172.21.15.115:0/1515974622 conn(0x7fcfa40f0b70 msgr2=0x7fcfa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 -- 172.21.15.115:0/1515974622 shutdown_connections 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.034+0000 7fcfabd71700 1 -- 172.21.15.115:0/1515974622 wait complete. 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfabd71700 1 Processor -- start 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfabd71700 1 -- start start 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfabd71700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:19.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfabd71700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4060280 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfa9b0d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfa9b0d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:37390/0 (socket says 172.21.15.115:37390) 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfa9b0d700 1 -- 172.21.15.115:0/300111895 learned_addr learned my addr 172.21.15.115:0/300111895 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfa9b0d700 1 -- 172.21.15.115:0/300111895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94005040 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.035+0000 7fcfa9b0d700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcf94013510 tx=0x7fcf9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf94009320 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa4060480 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf94013f00 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf94014d90 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa4109b20 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf940263a0 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.036+0000 7fcf9a7fc700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.037+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf94040fd0 con 0x7fcfa40f5410 2024-01-21T07:37:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.037+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa4045ad0 con 0x7fcfa40f5410 2024-01-21T07:37:19.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.037+0000 7fcfa930c700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:19.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.037+0000 7fcfa930c700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fcfa00096a0 tx=0x7fcfa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:19.042 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.041+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf94012920 con 0x7fcfa40f5410 2024-01-21T07:37:19.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.247+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfa410a240 con 0x7fcfa40f5410 2024-01-21T07:37:19.248 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.247+0000 7fcf9a7fc700 1 -- 172.21.15.115:0/300111895 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcf9402b020 con 0x7fcfa40f5410 2024-01-21T07:37:19.249 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:19.249 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:19.250 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 msgr2=0x7fcf90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:19.250 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fcfa00096a0 tx=0x7fcfa0006b40).stop 2024-01-21T07:37:19.250 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 msgr2=0x7fcfa41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:19.250 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcf94013510 tx=0x7fcf9400b040).stop 2024-01-21T07:37:19.250 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 shutdown_connections 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 --2- 172.21.15.115:0/300111895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfa40f5410 0x7fcfa41093f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 >> 172.21.15.115:0/300111895 conn(0x7fcfa40f0b70 msgr2=0x7fcfa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 shutdown_connections 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:19.249+0000 7fcfabd71700 1 -- 172.21.15.115:0/300111895 wait complete. 2024-01-21T07:37:19.251 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:19.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:19 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:19.247935+0000 mon.smithi107 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.115:0/300111895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:20.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:20 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:19.888713+0000 mgr.smithi107.adejht (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:20.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:20.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.759+0000 7f98b70c1700 1 -- 172.21.15.115:0/1641592099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 msgr2=0x7f98b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.759+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1641592099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b00f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f98a4009fb0 tx=0x7f98a4009b30).stop 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- 172.21.15.115:0/1641592099 shutdown_connections 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1641592099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b00f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- 172.21.15.115:0/1641592099 >> 172.21.15.115:0/1641592099 conn(0x7f98b00f0b70 msgr2=0x7f98b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:21.761 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- 172.21.15.115:0/1641592099 shutdown_connections 2024-01-21T07:37:21.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- 172.21.15.115:0/1641592099 wait complete. 2024-01-21T07:37:21.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 Processor -- start 2024-01-21T07:37:21.762 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- start start 2024-01-21T07:37:21.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:21.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.760+0000 7f98b70c1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b0106c00 con 0x7f98b00f5410 2024-01-21T07:37:21.763 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b4e5d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b4e5d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53942/0 (socket says 172.21.15.115:53942) 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b4e5d700 1 -- 172.21.15.115:0/1167449540 learned_addr learned my addr 172.21.15.115:0/1167449540 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b4e5d700 1 -- 172.21.15.115:0/1167449540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4005040 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b4e5d700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f98a4014000 tx=0x7f98a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a40095f0 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b0106e00 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b0107290 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98a4007e70 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.761+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a4014e50 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.762+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98a4025070 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.762+0000 7f98ad7fa700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.762+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b0045ad0 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.762+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98a4040d20 con 0x7f98b00f5410 2024-01-21T07:37:21.764 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.762+0000 7f98affff700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:21.765 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.763+0000 7f98affff700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 0x7f98a0032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f989c0096a0 tx=0x7f989c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:21.768 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.767+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98a40254e0 con 0x7f98b00f5410 2024-01-21T07:37:21.975 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.974+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98b0108090 con 0x7f98b00f5410 2024-01-21T07:37:21.976 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.975+0000 7f98ad7fa700 1 -- 172.21.15.115:0/1167449540 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98a401cbb0 con 0x7f98b00f5410 2024-01-21T07:37:21.976 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:21.976 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.976+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 msgr2=0x7f98a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.976+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 0x7f98a0032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f989c0096a0 tx=0x7f989c006b40).stop 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.976+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 msgr2=0x7f98b0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.976+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f98a4014000 tx=0x7f98a400b040).stop 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 shutdown_connections 2024-01-21T07:37:21.978 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:21.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 --2- 172.21.15.115:0/1167449540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b00f5410 0x7f98b0106710 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:21.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 >> 172.21.15.115:0/1167449540 conn(0x7f98b00f0b70 msgr2=0x7f98b005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:21.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 shutdown_connections 2024-01-21T07:37:21.979 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:21.977+0000 7f98b70c1700 1 -- 172.21.15.115:0/1167449540 wait complete. 2024-01-21T07:37:21.979 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:21.889065+0000 mgr.smithi107.adejht (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:23.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:22 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:21.975348+0000 mon.smithi107 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.115:0/1167449540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:23.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:23.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:24.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:23 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:23.889403+0000 mgr.smithi107.adejht (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:24.527 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.525+0000 7fca6886e700 1 -- 172.21.15.115:0/954198928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 msgr2=0x7fca640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:24.527 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.525+0000 7fca6886e700 1 --2- 172.21.15.115:0/954198928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca640f55c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fca54009fb0 tx=0x7fca54009b30).stop 2024-01-21T07:37:24.527 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- 172.21.15.115:0/954198928 shutdown_connections 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 --2- 172.21.15.115:0/954198928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca640f55c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- 172.21.15.115:0/954198928 >> 172.21.15.115:0/954198928 conn(0x7fca640f09a0 msgr2=0x7fca640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- 172.21.15.115:0/954198928 shutdown_connections 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- 172.21.15.115:0/954198928 wait complete. 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 Processor -- start 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- start start 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:24.528 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.526+0000 7fca6886e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca64106fc0 con 0x7fca640f51a0 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca637fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca637fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53948/0 (socket says 172.21.15.115:53948) 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca637fe700 1 -- 172.21.15.115:0/2352601583 learned_addr learned my addr 172.21.15.115:0/2352601583 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca637fe700 1 -- 172.21.15.115:0/2352601583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54005040 con 0x7fca640f51a0 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca637fe700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fca54013510 tx=0x7fca5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:24.530 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca54009320 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca54013f00 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca64060220 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca54014dc0 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.527+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6410bac0 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.528+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca540095f0 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.528+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca64045ad0 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.528+0000 7fca4bfff700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 0x7fca4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.528+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca540410c0 con 0x7fca640f51a0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.528+0000 7fca62ffd700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 0x7fca4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:24.531 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.529+0000 7fca62ffd700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 0x7fca4c032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fca580096a0 tx=0x7fca58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:24.534 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.533+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca5401c790 con 0x7fca640f51a0 2024-01-21T07:37:24.741 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.740+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca6410c280 con 0x7fca640f51a0 2024-01-21T07:37:24.742 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.740+0000 7fca4bfff700 1 -- 172.21.15.115:0/2352601583 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca54010230 con 0x7fca640f51a0 2024-01-21T07:37:24.742 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:24.742 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:24.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 msgr2=0x7fca4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:24.744 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 0x7fca4c032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fca580096a0 tx=0x7fca58006b40).stop 2024-01-21T07:37:24.745 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 msgr2=0x7fca6410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:24.745 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fca54013510 tx=0x7fca5400b040).stop 2024-01-21T07:37:24.745 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 shutdown_connections 2024-01-21T07:37:24.745 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fca4c02ff60 0x7fca4c032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:24.745 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 --2- 172.21.15.115:0/2352601583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca640f51a0 0x7fca6410b5d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:24.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 >> 172.21.15.115:0/2352601583 conn(0x7fca640f09a0 msgr2=0x7fca640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:24.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 shutdown_connections 2024-01-21T07:37:24.746 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:24.742+0000 7fca6886e700 1 -- 172.21.15.115:0/2352601583 wait complete. 2024-01-21T07:37:24.746 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:25.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:24 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:24.740800+0000 mon.smithi107 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.115:0/2352601583' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:26.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:26.130 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:26.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:25 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:25.889691+0000 mgr.smithi107.adejht (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:27.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 -- 172.21.15.115:0/2815088204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 msgr2=0x7f212c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:27.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 --2- 172.21.15.115:0/2815088204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f2124009960 tx=0x7f2124004d10).stop 2024-01-21T07:37:27.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 -- 172.21.15.115:0/2815088204 shutdown_connections 2024-01-21T07:37:27.293 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 --2- 172.21.15.115:0/2815088204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 -- 172.21.15.115:0/2815088204 >> 172.21.15.115:0/2815088204 conn(0x7f212c0f0b70 msgr2=0x7f212c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 -- 172.21.15.115:0/2815088204 shutdown_connections 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.290+0000 7f21311d5700 1 -- 172.21.15.115:0/2815088204 wait complete. 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f21311d5700 1 Processor -- start 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f21311d5700 1 -- start start 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f21311d5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f21311d5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c060220 con 0x7f212c0f5410 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f212ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f212ad9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:53956/0 (socket says 172.21.15.115:53956) 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.291+0000 7f212ad9d700 1 -- 172.21.15.115:0/2269412609 learned_addr learned my addr 172.21.15.115:0/2269412609 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f212ad9d700 1 -- 172.21.15.115:0/2269412609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124009640 con 0x7f212c0f5410 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f212ad9d700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2124004a10 tx=0x7f2124004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:27.294 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2124018070 con 0x7f212c0f5410 2024-01-21T07:37:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2124007e70 con 0x7f212c0f5410 2024-01-21T07:37:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c060420 con 0x7f212c0f5410 2024-01-21T07:37:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f212401d420 con 0x7f212c0f5410 2024-01-21T07:37:27.295 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.292+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c109b10 con 0x7f212c0f5410 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2124013630 con 0x7f212c0f5410 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c059100 con 0x7f212c0f5410 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f21137fe700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2124041580 con 0x7f212c0f5410 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f212a59c700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:27.296 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.293+0000 7f212a59c700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 0x7f2114032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f21200096a0 tx=0x7f2120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:27.299 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.298+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f212401b9c0 con 0x7f212c0f5410 2024-01-21T07:37:27.510 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.509+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f212c045ad0 con 0x7f212c0f5410 2024-01-21T07:37:27.511 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.510+0000 7f21137fe700 1 -- 172.21.15.115:0/2269412609 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f212401a2d0 con 0x7f212c0f5410 2024-01-21T07:37:27.511 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:27.511 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 msgr2=0x7f2114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 0x7f2114032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f21200096a0 tx=0x7f2120006b40).stop 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 msgr2=0x7f212c109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f2124004a10 tx=0x7f2124004d10).stop 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 shutdown_connections 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 --2- 172.21.15.115:0/2269412609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f212c0f5410 0x7f212c109460 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:27.513 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 >> 172.21.15.115:0/2269412609 conn(0x7f212c0f0b70 msgr2=0x7f212c0f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:27.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 shutdown_connections 2024-01-21T07:37:27.514 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:27.511+0000 7f21311d5700 1 -- 172.21.15.115:0/2269412609 wait complete. 2024-01-21T07:37:27.514 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:28.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:27 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:27.510070+0000 mon.smithi107 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.115:0/2269412609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:28.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:28.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:29.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:28 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:27.889953+0000 mgr.smithi107.adejht (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 -- 172.21.15.115:0/3076569319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 msgr2=0x7f05f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 --2- 172.21.15.115:0/3076569319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f40f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f05e4009fb0 tx=0x7f05e4009b30).stop 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 -- 172.21.15.115:0/3076569319 shutdown_connections 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 --2- 172.21.15.115:0/3076569319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f40f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 -- 172.21.15.115:0/3076569319 >> 172.21.15.115:0/3076569319 conn(0x7f05f40f0b70 msgr2=0x7f05f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:30.058 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 -- 172.21.15.115:0/3076569319 shutdown_connections 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.055+0000 7f05fc709700 1 -- 172.21.15.115:0/3076569319 wait complete. 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fc709700 1 Processor -- start 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fc709700 1 -- start start 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fc709700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fc709700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f4060240 con 0x7f05f40f5410 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fa4a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fa4a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34432/0 (socket says 172.21.15.115:34432) 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fa4a5700 1 -- 172.21.15.115:0/438664626 learned_addr learned my addr 172.21.15.115:0/438664626 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fa4a5700 1 -- 172.21.15.115:0/438664626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4005040 con 0x7f05f40f5410 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.056+0000 7f05fa4a5700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f05e4013510 tx=0x7f05e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05e4009320 con 0x7f05f40f5410 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f4060440 con 0x7f05f40f5410 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05e4013f00 con 0x7f05f40f5410 2024-01-21T07:37:30.059 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05e4014d90 con 0x7f05f40f5410 2024-01-21T07:37:30.060 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f4109ae0 con 0x7f05f40f5410 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05e40095f0 con 0x7f05f40f5410 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.057+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f4045ad0 con 0x7f05f40f5410 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.058+0000 7f05eaffd700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 0x7f05e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.058+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05e4040fe0 con 0x7f05f40f5410 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.058+0000 7f05f9ca4700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 0x7f05e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:30.061 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.058+0000 7f05f9ca4700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 0x7f05e0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f05f00096a0 tx=0x7f05f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:30.063 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.062+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05e401d7f0 con 0x7f05f40f5410 2024-01-21T07:37:30.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.270+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05f40fae60 con 0x7f05f40f5410 2024-01-21T07:37:30.272 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.270+0000 7f05eaffd700 1 -- 172.21.15.115:0/438664626 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f05e4010230 con 0x7f05f40f5410 2024-01-21T07:37:30.272 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:30.272 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 msgr2=0x7f05e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 0x7f05e0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f05f00096a0 tx=0x7f05f0006b40).stop 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 msgr2=0x7f05f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f05e4013510 tx=0x7f05e400b040).stop 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 shutdown_connections 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f05e002ff60 0x7f05e0032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 --2- 172.21.15.115:0/438664626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05f40f5410 0x7f05f4109430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 >> 172.21.15.115:0/438664626 conn(0x7f05f40f0b70 msgr2=0x7f05f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 shutdown_connections 2024-01-21T07:37:30.274 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:30.272+0000 7f05fc709700 1 -- 172.21.15.115:0/438664626 wait complete. 2024-01-21T07:37:30.275 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:30.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:29 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:29.890267+0000 mgr.smithi107.adejht (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:31.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:30 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:30.271038+0000 mon.smithi107 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.115:0/438664626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:31.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:31.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- 172.21.15.115:0/2205051245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 msgr2=0x7ff86c0f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 --2- 172.21.15.115:0/2205051245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c0f4ca0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7ff860009fb0 tx=0x7ff860009b30).stop 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- 172.21.15.115:0/2205051245 shutdown_connections 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 --2- 172.21.15.115:0/2205051245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c0f4ca0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- 172.21.15.115:0/2205051245 >> 172.21.15.115:0/2205051245 conn(0x7ff86c0f02e0 msgr2=0x7ff86c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- 172.21.15.115:0/2205051245 shutdown_connections 2024-01-21T07:37:32.824 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- 172.21.15.115:0/2205051245 wait complete. 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 Processor -- start 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- start start 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.819+0000 7ff87174a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c106ba0 con 0x7ff86c0f4880 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff86affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff86affd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34456/0 (socket says 172.21.15.115:34456) 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff86affd700 1 -- 172.21.15.115:0/342321266 learned_addr learned my addr 172.21.15.115:0/342321266 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff86affd700 1 -- 172.21.15.115:0/342321266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860005040 con 0x7ff86c0f4880 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff86affd700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff860013510 tx=0x7ff86000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff860009320 con 0x7ff86c0f4880 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c106da0 con 0x7ff86c0f4880 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff860013f00 con 0x7ff86c0f4880 2024-01-21T07:37:32.825 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff860014dc0 con 0x7ff86c0f4880 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.820+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c107230 con 0x7ff86c0f4880 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.821+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff860014420 con 0x7ff86c0f4880 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.821+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c045ad0 con 0x7ff86c0f4880 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.821+0000 7ff8537fe700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.821+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8600410d0 con 0x7ff86c0f4880 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.821+0000 7ff86a7fc700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:32.826 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.822+0000 7ff86a7fc700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 0x7ff854032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff85c0096a0 tx=0x7ff85c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:32.828 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:32.826+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8600095f0 con 0x7ff86c0f4880 2024-01-21T07:37:33.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.035+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff86c0fbf50 con 0x7ff86c0f4880 2024-01-21T07:37:33.038 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.036+0000 7ff8537fe700 1 -- 172.21.15.115:0/342321266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff86001cb10 con 0x7ff86c0f4880 2024-01-21T07:37:33.039 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:33.039 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:33.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 msgr2=0x7ff854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:33.040 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 0x7ff854032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff85c0096a0 tx=0x7ff85c006b40).stop 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 msgr2=0x7ff86c1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff860013510 tx=0x7ff86000b040).stop 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 shutdown_connections 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 --2- 172.21.15.115:0/342321266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff86c0f4880 0x7ff86c1066b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 >> 172.21.15.115:0/342321266 conn(0x7ff86c0f02e0 msgr2=0x7ff86c0f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 shutdown_connections 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:33.038+0000 7ff87174a700 1 -- 172.21.15.115:0/342321266 wait complete. 2024-01-21T07:37:33.041 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:33.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:32 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:31.890572+0000 mgr.smithi107.adejht (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:34.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:33 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:33.037767+0000 mon.smithi107 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.115:0/342321266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:34.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:34.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:35.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:34 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:33.890886+0000 mgr.smithi107.adejht (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:35.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 -- 172.21.15.115:0/1641947244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2880f51a0 msgr2=0x7fa2880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:35.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 --2- 172.21.15.115:0/1641947244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2880f51a0 0x7fa2880f55c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fa27c009fb0 tx=0x7fa27c009b30).stop 2024-01-21T07:37:35.581 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 -- 172.21.15.115:0/1641947244 shutdown_connections 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 --2- 172.21.15.115:0/1641947244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2880f51a0 0x7fa2880f55c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 -- 172.21.15.115:0/1641947244 >> 172.21.15.115:0/1641947244 conn(0x7fa2880f09a0 msgr2=0x7fa2880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 -- 172.21.15.115:0/1641947244 shutdown_connections 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.578+0000 7fa28d280700 1 -- 172.21.15.115:0/1641947244 wait complete. 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.579+0000 7fa28d280700 1 Processor -- start 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.579+0000 7fa28d280700 1 -- start start 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.579+0000 7fa28d280700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:35.582 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.579+0000 7fa28d280700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa288105e30 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa287fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa287fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34470/0 (socket says 172.21.15.115:34470) 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa287fff700 1 -- 172.21.15.115:0/1792574546 learned_addr learned my addr 172.21.15.115:0/1792574546 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa287fff700 1 -- 172.21.15.115:0/1792574546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c005040 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa287fff700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa27c009f80 tx=0x7fa27c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27c0136c0 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa288106030 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa27c0095f0 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27c014420 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.580+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa288106b80 con 0x7fa288105520 2024-01-21T07:37:35.584 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.581+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa27c025060 con 0x7fa288105520 2024-01-21T07:37:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.581+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa288059100 con 0x7fa288105520 2024-01-21T07:37:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.581+0000 7fa284ff9700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 0x7fa270034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.581+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa27c040dd0 con 0x7fa288105520 2024-01-21T07:37:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.581+0000 7fa2877fe700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 0x7fa270034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.582+0000 7fa2877fe700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 0x7fa270034630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa2780096a0 tx=0x7fa278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:35.588 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.586+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa27c004490 con 0x7fa288105520 2024-01-21T07:37:35.794 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.792+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa288045ad0 con 0x7fa288105520 2024-01-21T07:37:35.795 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.793+0000 7fa284ff9700 1 -- 172.21.15.115:0/1792574546 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa27c02d2d0 con 0x7fa288105520 2024-01-21T07:37:35.795 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:35.795 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 msgr2=0x7fa270034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 0x7fa270034630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa2780096a0 tx=0x7fa278006b40).stop 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 msgr2=0x7fa288105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fa27c009f80 tx=0x7fa27c009b30).stop 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 shutdown_connections 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fa270032170 0x7fa270034630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 --2- 172.21.15.115:0/1792574546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa288105520 0x7fa288105940 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 >> 172.21.15.115:0/1792574546 conn(0x7fa2880f09a0 msgr2=0x7fa2880f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 shutdown_connections 2024-01-21T07:37:35.797 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:35.794+0000 7fa28d280700 1 -- 172.21.15.115:0/1792574546 wait complete. 2024-01-21T07:37:35.798 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:36.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:35 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:35.794341+0000 mon.smithi107 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.115:0/1792574546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:37.127 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:36 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:35.891263+0000 mgr.smithi107.adejht (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:37.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:37.185 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:38.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:37 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:37.891589+0000 mgr.smithi107.adejht (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:38.324 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 -- 172.21.15.115:0/1566644729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 msgr2=0x7f33d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1566644729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d40f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f33c8009fb0 tx=0x7f33c8009b30).stop 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 -- 172.21.15.115:0/1566644729 shutdown_connections 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1566644729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d40f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 -- 172.21.15.115:0/1566644729 >> 172.21.15.115:0/1566644729 conn(0x7f33d40f0b70 msgr2=0x7f33d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 -- 172.21.15.115:0/1566644729 shutdown_connections 2024-01-21T07:37:38.325 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.322+0000 7f33d8b58700 1 -- 172.21.15.115:0/1566644729 wait complete. 2024-01-21T07:37:38.327 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d8b58700 1 Processor -- start 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d8b58700 1 -- start start 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d8b58700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d8b58700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4105af0 con 0x7f33d40f5410 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:34498/0 (socket says 172.21.15.115:34498) 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.323+0000 7f33d259c700 1 -- 172.21.15.115:0/1953215783 learned_addr learned my addr 172.21.15.115:0/1953215783 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33d259c700 1 -- 172.21.15.115:0/1953215783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8005040 con 0x7f33d40f5410 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33d259c700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f33c8013510 tx=0x7f33c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8009320 con 0x7f33d40f5410 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d4105cf0 con 0x7f33d40f5410 2024-01-21T07:37:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d4106180 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c8013f00 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.324+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8014ce0 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.325+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33c8014420 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.325+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b8004fa0 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.325+0000 7f33c2ffd700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 0x7f33bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.325+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33c8040ed0 con 0x7f33d40f5410 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.325+0000 7f33d1d9b700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 0x7f33bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:38.329 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.326+0000 7f33d1d9b700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 0x7f33bc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f33c40096a0 tx=0x7f33c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:38.332 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.330+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c8014e40 con 0x7f33d40f5410 2024-01-21T07:37:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.538+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33b8005670 con 0x7f33d40f5410 2024-01-21T07:37:38.541 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.539+0000 7f33c2ffd700 1 -- 172.21.15.115:0/1953215783 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33c8012750 con 0x7f33d40f5410 2024-01-21T07:37:38.541 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:38.541 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 msgr2=0x7f33bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 0x7f33bc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f33c40096a0 tx=0x7f33c4006b40).stop 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 msgr2=0x7f33d4105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f33c8013510 tx=0x7f33c800b040).stop 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 shutdown_connections 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f33bc02ff60 0x7f33bc032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 --2- 172.21.15.115:0/1953215783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33d40f5410 0x7f33d4105600 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 >> 172.21.15.115:0/1953215783 conn(0x7f33d40f0b70 msgr2=0x7f33d40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 shutdown_connections 2024-01-21T07:37:38.543 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:38.540+0000 7f33d8b58700 1 -- 172.21.15.115:0/1953215783 wait complete. 2024-01-21T07:37:38.544 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:39.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:38 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:38.540173+0000 mon.smithi107 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.115:0/1953215783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:39.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:39.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:40.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:39 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:39.891889+0000 mgr.smithi107.adejht (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:41.050 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- 172.21.15.115:0/2064122886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 msgr2=0x7f2a080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/2064122886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a080f55c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f29fc009fb0 tx=0x7f29fc009b30).stop 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- 172.21.15.115:0/2064122886 shutdown_connections 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/2064122886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a080f55c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- 172.21.15.115:0/2064122886 >> 172.21.15.115:0/2064122886 conn(0x7f2a080f09a0 msgr2=0x7f2a080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- 172.21.15.115:0/2064122886 shutdown_connections 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- 172.21.15.115:0/2064122886 wait complete. 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 Processor -- start 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.048+0000 7f2a0dfea700 1 -- start start 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0dfea700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:41.051 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0dfea700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08060220 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0cfe8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0cfe8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59334/0 (socket says 172.21.15.115:59334) 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0cfe8700 1 -- 172.21.15.115:0/31167074 learned_addr learned my addr 172.21.15.115:0/31167074 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0cfe8700 1 -- 172.21.15.115:0/31167074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc005040 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0cfe8700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f29fc013510 tx=0x7f29fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc009320 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a08060420 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29fc013f00 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc014dc0 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.049+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a08109880 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.050+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29fc014420 con 0x7f2a080f51a0 2024-01-21T07:37:41.053 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.050+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a08059100 con 0x7f2a080f51a0 2024-01-21T07:37:41.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.050+0000 7f2a057fa700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 0x7f29f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:41.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.050+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29fc041050 con 0x7f2a080f51a0 2024-01-21T07:37:41.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.051+0000 7f2a07fff700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 0x7f29f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:41.054 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.051+0000 7f2a07fff700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 0x7f29f8032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f29f40096a0 tx=0x7f29f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:41.057 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.056+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29fc0095f0 con 0x7f2a080f51a0 2024-01-21T07:37:41.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.265+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a08045ad0 con 0x7f2a080f51a0 2024-01-21T07:37:41.267 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.265+0000 7f2a057fa700 1 -- 172.21.15.115:0/31167074 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29fc01cb10 con 0x7f2a080f51a0 2024-01-21T07:37:41.268 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:41.268 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 msgr2=0x7f29f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 0x7f29f8032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f29f40096a0 tx=0x7f29f4006b40).stop 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 msgr2=0x7f2a081091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f29fc013510 tx=0x7f29fc00b040).stop 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 shutdown_connections 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f29f802ff60 0x7f29f8032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 --2- 172.21.15.115:0/31167074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a080f51a0 0x7f2a081091d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 >> 172.21.15.115:0/31167074 conn(0x7f2a080f09a0 msgr2=0x7f2a080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 shutdown_connections 2024-01-21T07:37:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:41.267+0000 7f2a0dfea700 1 -- 172.21.15.115:0/31167074 wait complete. 2024-01-21T07:37:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:41.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:41 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:41.266824+0000 mon.smithi107 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.115:0/31167074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:42.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:42.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:43.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:42 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:41.892325+0000 mgr.smithi107.adejht (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 -- 172.21.15.115:0/3582521499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 msgr2=0x7f7ec00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 --2- 172.21.15.115:0/3582521499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec00f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7eb4009fb0 tx=0x7f7eb4009b30).stop 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 -- 172.21.15.115:0/3582521499 shutdown_connections 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 --2- 172.21.15.115:0/3582521499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec00f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 -- 172.21.15.115:0/3582521499 >> 172.21.15.115:0/3582521499 conn(0x7f7ec00f0b70 msgr2=0x7f7ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 -- 172.21.15.115:0/3582521499 shutdown_connections 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.795+0000 7f7ec6993700 1 -- 172.21.15.115:0/3582521499 wait complete. 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ec6993700 1 Processor -- start 2024-01-21T07:37:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ec6993700 1 -- start start 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ec6993700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ec6993700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0060220 con 0x7f7ec00f5410 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ebffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ebffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59358/0 (socket says 172.21.15.115:59358) 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ebffff700 1 -- 172.21.15.115:0/2027340319 learned_addr learned my addr 172.21.15.115:0/2027340319 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.796+0000 7f7ebffff700 1 -- 172.21.15.115:0/2027340319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4005040 con 0x7f7ec00f5410 2024-01-21T07:37:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ebffff700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f7eb4014000 tx=0x7f7eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb40095f0 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7eb4007e70 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec0060420 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb4014ea0 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.797+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec0109aa0 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7eb4025070 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ebcff9700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 0x7f7ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7eb4040990 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0059100 con 0x7f7ec00f5410 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ebf7fe700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 0x7f7ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.798+0000 7f7ebf7fe700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 0x7f7ea8032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7eb00096a0 tx=0x7f7eb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:43.804 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:43.803+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7eb4026350 con 0x7f7ec00f5410 2024-01-21T07:37:44.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.008+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ec0045ad0 con 0x7f7ec00f5410 2024-01-21T07:37:44.010 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.008+0000 7f7ebcff9700 1 -- 172.21.15.115:0/2027340319 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ec0045ad0 con 0x7f7ec00f5410 2024-01-21T07:37:44.011 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:44.011 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 msgr2=0x7f7ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 0x7f7ea8032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7eb00096a0 tx=0x7f7eb0006b40).stop 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 msgr2=0x7f7ec01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f7eb4014000 tx=0x7f7eb400b040).stop 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 shutdown_connections 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f7ea802ff60 0x7f7ea8032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 --2- 172.21.15.115:0/2027340319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec00f5410 0x7f7ec01093f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.010+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 >> 172.21.15.115:0/2027340319 conn(0x7f7ec00f0b70 msgr2=0x7f7ec00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.011+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 shutdown_connections 2024-01-21T07:37:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:44.011+0000 7f7ec6993700 1 -- 172.21.15.115:0/2027340319 wait complete. 2024-01-21T07:37:44.014 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:44.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:43 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:43.892733+0000 mgr.smithi107.adejht (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:45.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:44 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:44.009926+0000 mon.smithi107 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.115:0/2027340319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:45.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:45.387 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:46.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:45 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:45.893126+0000 mgr.smithi107.adejht (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 -- 172.21.15.115:0/3235605446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 msgr2=0x7fa0000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 --2- 172.21.15.115:0/3235605446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa0000f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9ff0009fb0 tx=0x7f9ff0009b30).stop 2024-01-21T07:37:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 -- 172.21.15.115:0/3235605446 shutdown_connections 2024-01-21T07:37:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 --2- 172.21.15.115:0/3235605446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa0000f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:46.562 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 -- 172.21.15.115:0/3235605446 >> 172.21.15.115:0/3235605446 conn(0x7fa0000f0b70 msgr2=0x7fa0000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 -- 172.21.15.115:0/3235605446 shutdown_connections 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.558+0000 7fa0052b6700 1 -- 172.21.15.115:0/3235605446 wait complete. 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7fa0052b6700 1 Processor -- start 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7fa0052b6700 1 -- start start 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7fa0052b6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7fa0052b6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa000107210 con 0x7fa0000f5410 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7f9ffeffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7f9ffeffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59380/0 (socket says 172.21.15.115:59380) 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7f9ffeffd700 1 -- 172.21.15.115:0/2644834276 learned_addr learned my addr 172.21.15.115:0/2644834276 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.559+0000 7f9ffeffd700 1 -- 172.21.15.115:0/2644834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0005040 con 0x7fa0000f5410 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7f9ffeffd700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9ff0013510 tx=0x7f9ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff0009320 con 0x7fa0000f5410 2024-01-21T07:37:46.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ff0013f00 con 0x7fa0000f5410 2024-01-21T07:37:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff0014d90 con 0x7fa0000f5410 2024-01-21T07:37:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa000060220 con 0x7fa0000f5410 2024-01-21T07:37:46.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.560+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00010bd10 con 0x7fa0000f5410 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ff00095f0 con 0x7fa0000f5410 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa000045ad0 con 0x7fa0000f5410 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7f9fe77fe700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 0x7f9fe8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ff0041000 con 0x7fa0000f5410 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7f9ffe7fc700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 0x7f9fe8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:46.565 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.561+0000 7f9ffe7fc700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 0x7f9fe8032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ff40096a0 tx=0x7f9ff4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.566+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ff001d7f0 con 0x7fa0000f5410 2024-01-21T07:37:46.771 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.769+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0000fae60 con 0x7fa0000f5410 2024-01-21T07:37:46.772 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.770+0000 7f9fe77fe700 1 -- 172.21.15.115:0/2644834276 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ff0010230 con 0x7fa0000f5410 2024-01-21T07:37:46.772 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:46.772 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 msgr2=0x7f9fe8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 0x7f9fe8032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ff40096a0 tx=0x7f9ff4006b40).stop 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 msgr2=0x7fa00010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9ff0013510 tx=0x7f9ff000b040).stop 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 shutdown_connections 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f9fe802ff60 0x7f9fe8032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 --2- 172.21.15.115:0/2644834276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0000f5410 0x7fa00010b820 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 >> 172.21.15.115:0/2644834276 conn(0x7fa0000f0b70 msgr2=0x7fa0000f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 shutdown_connections 2024-01-21T07:37:46.774 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:46.771+0000 7fa0052b6700 1 -- 172.21.15.115:0/2644834276 wait complete. 2024-01-21T07:37:46.775 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:47.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:46 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:46.771426+0000 mon.smithi107 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.115:0/2644834276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:48.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:48.171 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:48.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:47 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:47.893624+0000 mgr.smithi107.adejht (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:49.381 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 -- 172.21.15.115:0/1781780663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 msgr2=0x7faff00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 --2- 172.21.15.115:0/1781780663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff00f90f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fafe4009fb0 tx=0x7fafe4009b30).stop 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 -- 172.21.15.115:0/1781780663 shutdown_connections 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 --2- 172.21.15.115:0/1781780663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff00f90f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 -- 172.21.15.115:0/1781780663 >> 172.21.15.115:0/1781780663 conn(0x7faff00f0b70 msgr2=0x7faff00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 -- 172.21.15.115:0/1781780663 shutdown_connections 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.378+0000 7faff7037700 1 -- 172.21.15.115:0/1781780663 wait complete. 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.379+0000 7faff7037700 1 Processor -- start 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.379+0000 7faff7037700 1 -- start start 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.379+0000 7faff7037700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.379+0000 7faff7037700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff0108de0 con 0x7faff0060280 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff4dd3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff4dd3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59000/0 (socket says 172.21.15.115:59000) 2024-01-21T07:37:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff4dd3700 1 -- 172.21.15.115:0/2519558611 learned_addr learned my addr 172.21.15.115:0/2519558611 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff4dd3700 1 -- 172.21.15.115:0/2519558611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe4005040 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff4dd3700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fafe4009830 tx=0x7fafe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe4009320 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafe40136c0 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe401ee90 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff0108fe0 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.380+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff0109470 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.381+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafe401e420 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.381+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff0045ad0 con 0x7faff0060280 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.381+0000 7fafed7fa700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 0x7fafe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.381+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafe4040fa0 con 0x7faff0060280 2024-01-21T07:37:49.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.381+0000 7fafeffff700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 0x7fafe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:49.385 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.382+0000 7fafeffff700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 0x7fafe00323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:49.388 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.386+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafe40095f0 con 0x7faff0060280 2024-01-21T07:37:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.587+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faff00fbf50 con 0x7faff0060280 2024-01-21T07:37:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.588+0000 7fafed7fa700 1 -- 172.21.15.115:0/2519558611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafe402c2d0 con 0x7faff0060280 2024-01-21T07:37:49.591 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:49.591 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 msgr2=0x7fafe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 0x7fafe00323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).stop 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 msgr2=0x7faff01088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fafe4009830 tx=0x7fafe400b040).stop 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 shutdown_connections 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7fafe002ff10 0x7fafe00323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 --2- 172.21.15.115:0/2519558611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff0060280 0x7faff01088f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 >> 172.21.15.115:0/2519558611 conn(0x7faff00f0b70 msgr2=0x7faff00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 shutdown_connections 2024-01-21T07:37:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:49.589+0000 7faff7037700 1 -- 172.21.15.115:0/2519558611 wait complete. 2024-01-21T07:37:49.593 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:50.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:49 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:49.589332+0000 mon.smithi107 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.115:0/2519558611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:50.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:50.970 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:51.065 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:50 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:49.894030+0000 mgr.smithi107.adejht (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- 172.21.15.115:0/4156779016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 msgr2=0x7f3a640f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 --2- 172.21.15.115:0/4156779016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a640f68a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f3a54009fb0 tx=0x7f3a54009b30).stop 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- 172.21.15.115:0/4156779016 shutdown_connections 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 --2- 172.21.15.115:0/4156779016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a640f68a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- 172.21.15.115:0/4156779016 >> 172.21.15.115:0/4156779016 conn(0x7f3a640f1c00 msgr2=0x7f3a640f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- 172.21.15.115:0/4156779016 shutdown_connections 2024-01-21T07:37:52.126 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- 172.21.15.115:0/4156779016 wait complete. 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 Processor -- start 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.122+0000 7f3a6be60700 1 -- start start 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a6be60700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a6be60700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64069e20 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a69bfc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a69bfc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59018/0 (socket says 172.21.15.115:59018) 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a69bfc700 1 -- 172.21.15.115:0/3030995392 learned_addr learned my addr 172.21.15.115:0/3030995392 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a69bfc700 1 -- 172.21.15.115:0/3030995392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54005040 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.123+0000 7f3a69bfc700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f3a54014000 tx=0x7f3a5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a540095f0 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a54007e70 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a64069fc0 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a54014ea0 con 0x7f3a640f6480 2024-01-21T07:37:52.127 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6406a450 con 0x7f3a640f6480 2024-01-21T07:37:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a54025070 con 0x7f3a640f6480 2024-01-21T07:37:52.128 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.124+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a64059100 con 0x7f3a640f6480 2024-01-21T07:37:52.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.125+0000 7f3a5a7fc700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 0x7f3a50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:52.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.125+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a54040970 con 0x7f3a640f6480 2024-01-21T07:37:52.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.125+0000 7f3a693fb700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 0x7f3a50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:52.129 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.125+0000 7f3a693fb700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 0x7f3a50032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3a600096a0 tx=0x7f3a60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:52.131 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.129+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a540079c0 con 0x7f3a640f6480 2024-01-21T07:37:52.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.338+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a640fbf40 con 0x7f3a640f6480 2024-01-21T07:37:52.340 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.338+0000 7f3a5a7fc700 1 -- 172.21.15.115:0/3030995392 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a5400f300 con 0x7f3a640f6480 2024-01-21T07:37:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:52.341 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:52.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 msgr2=0x7f3a50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:52.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 0x7f3a50032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3a600096a0 tx=0x7f3a60006b40).stop 2024-01-21T07:37:52.342 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 msgr2=0x7f3a6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f3a54014000 tx=0x7f3a5400b040).stop 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 shutdown_connections 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f3a5002ff60 0x7f3a50032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 --2- 172.21.15.115:0/3030995392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a640f6480 0x7f3a6406b9a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 >> 172.21.15.115:0/3030995392 conn(0x7f3a640f1c00 msgr2=0x7f3a640f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 shutdown_connections 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:52.340+0000 7f3a6be60700 1 -- 172.21.15.115:0/3030995392 wait complete. 2024-01-21T07:37:52.343 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:53.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:51.894396+0000 mgr.smithi107.adejht (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:53.316 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:52 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:52.339974+0000 mon.smithi107 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.115:0/3030995392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:53.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:53.723 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:54.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:53 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:53.894749+0000 mgr.smithi107.adejht (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 -- 172.21.15.115:0/3746275347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 msgr2=0x7f49e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:54.891 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 --2- 172.21.15.115:0/3746275347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e40f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f49d8009fb0 tx=0x7f49d8009b30).stop 2024-01-21T07:37:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 -- 172.21.15.115:0/3746275347 shutdown_connections 2024-01-21T07:37:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 --2- 172.21.15.115:0/3746275347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e40f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 -- 172.21.15.115:0/3746275347 >> 172.21.15.115:0/3746275347 conn(0x7f49e40f0b70 msgr2=0x7f49e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 -- 172.21.15.115:0/3746275347 shutdown_connections 2024-01-21T07:37:54.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.889+0000 7f49e956b700 1 -- 172.21.15.115:0/3746275347 wait complete. 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e956b700 1 Processor -- start 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e956b700 1 -- start start 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e956b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e956b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e4106c00 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e2ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e2ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59042/0 (socket says 172.21.15.115:59042) 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e2ffd700 1 -- 172.21.15.115:0/1194864910 learned_addr learned my addr 172.21.15.115:0/1194864910 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e2ffd700 1 -- 172.21.15.115:0/1194864910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8005040 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.890+0000 7f49e2ffd700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f49d8013510 tx=0x7f49d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.891+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49d8009320 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.891+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49d8013f00 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.891+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e4106e00 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.891+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49d8014d90 con 0x7f49e40f5410 2024-01-21T07:37:54.894 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.891+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e4107290 con 0x7f49e40f5410 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49d8014420 con 0x7f49e40f5410 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e4045ad0 con 0x7f49e40f5410 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49cb7fe700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 0x7f49cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49d8040fb0 con 0x7f49e40f5410 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49e27fc700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 0x7f49cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:54.895 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.892+0000 7f49e27fc700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 0x7f49cc032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f49d40096a0 tx=0x7f49d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:54.899 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:54.897+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49d80095f0 con 0x7f49e40f5410 2024-01-21T07:37:55.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.101+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49e4108140 con 0x7f49e40f5410 2024-01-21T07:37:55.104 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.102+0000 7f49cb7fe700 1 -- 172.21.15.115:0/1194864910 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f49d801db70 con 0x7f49e40f5410 2024-01-21T07:37:55.104 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:55.104 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 msgr2=0x7f49cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 0x7f49cc032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f49d40096a0 tx=0x7f49d4006b40).stop 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 msgr2=0x7f49e4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f49d8013510 tx=0x7f49d800b040).stop 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 shutdown_connections 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f49cc02ff60 0x7f49cc032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 --2- 172.21.15.115:0/1194864910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49e40f5410 0x7f49e4106710 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 >> 172.21.15.115:0/1194864910 conn(0x7f49e40f0b70 msgr2=0x7f49e405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 shutdown_connections 2024-01-21T07:37:55.106 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:55.104+0000 7f49e956b700 1 -- 172.21.15.115:0/1194864910 wait complete. 2024-01-21T07:37:55.107 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:55.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:55 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:55.103498+0000 mon.smithi107 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.115:0/1194864910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:56.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:56.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:56.565 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:56 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:55.895055+0000 mgr.smithi107.adejht (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:37:57.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 -- 172.21.15.115:0/3745273578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 msgr2=0x7f74880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:57.613 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 --2- 172.21.15.115:0/3745273578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f74880f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f747c009fb0 tx=0x7f747c009b30).stop 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 -- 172.21.15.115:0/3745273578 shutdown_connections 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 --2- 172.21.15.115:0/3745273578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f74880f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 -- 172.21.15.115:0/3745273578 >> 172.21.15.115:0/3745273578 conn(0x7f74880f0b70 msgr2=0x7f74880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 -- 172.21.15.115:0/3745273578 shutdown_connections 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.609+0000 7f748ca1d700 1 -- 172.21.15.115:0/3745273578 wait complete. 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748ca1d700 1 Processor -- start 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748ca1d700 1 -- start start 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748ca1d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748ca1d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488060240 con 0x7f74880f5410 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:59070/0 (socket says 172.21.15.115:59070) 2024-01-21T07:37:57.614 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.610+0000 7f748659c700 1 -- 172.21.15.115:0/743929024 learned_addr learned my addr 172.21.15.115:0/743929024 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f748659c700 1 -- 172.21.15.115:0/743929024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c005040 con 0x7f74880f5410 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f748659c700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f747c013510 tx=0x7f747c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c009320 con 0x7f74880f5410 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7488060440 con 0x7f74880f5410 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f747c013f00 con 0x7f74880f5410 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f747c014d90 con 0x7f74880f5410 2024-01-21T07:37:57.615 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.611+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7488109ae0 con 0x7f74880f5410 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.612+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f747c0095f0 con 0x7f74880f5410 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.612+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7488045ad0 con 0x7f74880f5410 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.612+0000 7f7476ffd700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 0x7f7470032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.612+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f747c040fe0 con 0x7f74880f5410 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.612+0000 7f7485d9b700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 0x7f7470032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:37:57.616 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.613+0000 7f7485d9b700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 0x7f7470032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:37:57.619 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.617+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f747c01d7f0 con 0x7f74880f5410 2024-01-21T07:37:57.827 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.824+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74880fae60 con 0x7f74880f5410 2024-01-21T07:37:57.827 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.825+0000 7f7476ffd700 1 -- 172.21.15.115:0/743929024 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f747c010230 con 0x7f74880f5410 2024-01-21T07:37:57.827 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:37:57.828 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 msgr2=0x7f7470032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 0x7f7470032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).stop 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 msgr2=0x7f7488109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f747c013510 tx=0x7f747c00b040).stop 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 shutdown_connections 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f747002ff60 0x7f7470032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 --2- 172.21.15.115:0/743929024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74880f5410 0x7f7488109430 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 >> 172.21.15.115:0/743929024 conn(0x7f74880f0b70 msgr2=0x7f74880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 shutdown_connections 2024-01-21T07:37:57.829 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:37:57.827+0000 7f748ca1d700 1 -- 172.21.15.115:0/743929024 wait complete. 2024-01-21T07:37:57.830 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:37:58.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:57 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:37:57.826599+0000 mon.smithi107 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.115:0/743929024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:37:59.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:37:59.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:37:59.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:58 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:57.895365+0000 mgr.smithi107.adejht (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:38:00.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:37:59 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:37:59.895620+0000 mgr.smithi107.adejht (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:38:00.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/3656307099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 msgr2=0x7f6cc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:00.348 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/3656307099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f6cbc009fb0 tx=0x7f6cbc009b30).stop 2024-01-21T07:38:00.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/3656307099 shutdown_connections 2024-01-21T07:38:00.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/3656307099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:00.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/3656307099 >> 172.21.15.115:0/3656307099 conn(0x7f6cc80f0b70 msgr2=0x7f6cc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:00.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/3656307099 shutdown_connections 2024-01-21T07:38:00.349 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.346+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/3656307099 wait complete. 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6ccd0ff700 1 Processor -- start 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6ccd0ff700 1 -- start start 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6ccd0ff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6ccd0ff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc80fb820 con 0x7f6cc80f5410 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6cc6d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6cc6d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48018/0 (socket says 172.21.15.115:48018) 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.347+0000 7f6cc6d9d700 1 -- 172.21.15.115:0/156061088 learned_addr learned my addr 172.21.15.115:0/156061088 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6cc6d9d700 1 -- 172.21.15.115:0/156061088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc005040 con 0x7f6cc80f5410 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6cc6d9d700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6cbc013510 tx=0x7f6cbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:00.351 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cbc009320 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc80fb9c0 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cbc013f00 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cbc014d90 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.348+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc80fbe50 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cbc0095f0 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc8059100 con 0x7f6cc80f5410 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6caf7fe700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 0x7f6cb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cbc041000 con 0x7f6cc80f5410 2024-01-21T07:38:00.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6cc659c700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 0x7f6cb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:00.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.349+0000 7f6cc659c700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 0x7f6cb0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6cb80096a0 tx=0x7f6cb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:00.356 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.354+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cbc014420 con 0x7f6cc80f5410 2024-01-21T07:38:00.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.559+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cc80fae60 con 0x7f6cc80f5410 2024-01-21T07:38:00.561 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.559+0000 7f6caf7fe700 1 -- 172.21.15.115:0/156061088 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6cbc010230 con 0x7f6cc80f5410 2024-01-21T07:38:00.562 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:38:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:38:00.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 msgr2=0x7f6cb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:00.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 0x7f6cb0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6cb80096a0 tx=0x7f6cb8006b40).stop 2024-01-21T07:38:00.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 msgr2=0x7f6cc80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:00.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6cbc013510 tx=0x7f6cbc00b040).stop 2024-01-21T07:38:00.563 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 shutdown_connections 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f6cb002ff60 0x7f6cb0032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 --2- 172.21.15.115:0/156061088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cc80f5410 0x7f6cc80fd3a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 >> 172.21.15.115:0/156061088 conn(0x7f6cc80f0b70 msgr2=0x7f6cc805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 shutdown_connections 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:00.561+0000 7f6ccd0ff700 1 -- 172.21.15.115:0/156061088 wait complete. 2024-01-21T07:38:00.564 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:38:01.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:00 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:38:00.560850+0000 mon.smithi107 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.115:0/156061088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:38:01.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:38:01.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:38:03.134 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 -- 172.21.15.115:0/3640856211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 msgr2=0x7f22140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 --2- 172.21.15.115:0/3640856211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f22140f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2204009fb0 tx=0x7f2204009b30).stop 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 -- 172.21.15.115:0/3640856211 shutdown_connections 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 --2- 172.21.15.115:0/3640856211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f22140f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 -- 172.21.15.115:0/3640856211 >> 172.21.15.115:0/3640856211 conn(0x7f22140f0b70 msgr2=0x7f22140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 -- 172.21.15.115:0/3640856211 shutdown_connections 2024-01-21T07:38:03.135 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.132+0000 7f221c4bf700 1 -- 172.21.15.115:0/3640856211 wait complete. 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221c4bf700 1 Processor -- start 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221c4bf700 1 -- start start 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221c4bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221c4bf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2214105b20 con 0x7f22140f5410 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221a25b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221a25b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48042/0 (socket says 172.21.15.115:48042) 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221a25b700 1 -- 172.21.15.115:0/2334894493 learned_addr learned my addr 172.21.15.115:0/2334894493 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:38:03.137 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.133+0000 7f221a25b700 1 -- 172.21.15.115:0/2334894493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204005040 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f221a25b700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2204013510 tx=0x7f220400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2204009320 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2204013f00 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2214105d20 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2204014d90 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.134+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141061b0 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22040095f0 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2214045ad0 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f220affd700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2204042d90 con 0x7f22140f5410 2024-01-21T07:38:03.138 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f2219a5a700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:03.139 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.135+0000 7f2219a5a700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f22100096a0 tx=0x7f2210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:03.141 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.140+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f220401f7f0 con 0x7f22140f5410 2024-01-21T07:38:03.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:02 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:38:01.895900+0000 mgr.smithi107.adejht (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:38:03.352 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.350+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22140fae60 con 0x7f22140f5410 2024-01-21T07:38:03.353 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f220affd700 1 -- 172.21.15.115:0/2334894493 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2204011290 con 0x7f22140f5410 2024-01-21T07:38:03.353 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:38:03.353 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:38:03.354 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 msgr2=0x7f2200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f22100096a0 tx=0x7f2210006b40).stop 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 msgr2=0x7f2214105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2204013510 tx=0x7f220400b040).stop 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 shutdown_connections 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 --2- 172.21.15.115:0/2334894493 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22140f5410 0x7f2214105630 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 >> 172.21.15.115:0/2334894493 conn(0x7f22140f0b70 msgr2=0x7f22140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 shutdown_connections 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:03.352+0000 7f221c4bf700 1 -- 172.21.15.115:0/2334894493 wait complete. 2024-01-21T07:38:03.355 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:38:04.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:03 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: audit 2024-01-21T07:38:03.352110+0000 mon.smithi107 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.115:0/2334894493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T07:38:04.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T07:38:04.734 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T07:38:05.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:04 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:38:03.896196+0000 mgr.smithi107.adejht (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:38:05.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- 172.21.15.115:0/3702697788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 msgr2=0x7f033c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:05.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 --2- 172.21.15.115:0/3702697788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c0f55c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f0330009fb0 tx=0x7f0330009b30).stop 2024-01-21T07:38:05.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- 172.21.15.115:0/3702697788 shutdown_connections 2024-01-21T07:38:05.886 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 --2- 172.21.15.115:0/3702697788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c0f55c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- 172.21.15.115:0/3702697788 >> 172.21.15.115:0/3702697788 conn(0x7f033c0f09a0 msgr2=0x7f033c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- 172.21.15.115:0/3702697788 shutdown_connections 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- 172.21.15.115:0/3702697788 wait complete. 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 Processor -- start 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.884+0000 7f03419db700 1 -- start start 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03419db700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03419db700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c108bc0 con 0x7f033c0f51a0 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03409d9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03409d9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.115:48052/0 (socket says 172.21.15.115:48052) 2024-01-21T07:38:05.887 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03409d9700 1 -- 172.21.15.115:0/2201842839 learned_addr learned my addr 172.21.15.115:0/2201842839 (peer_addr_for_me v2:172.21.15.115:0/0) 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.885+0000 7f03409d9700 1 -- 172.21.15.115:0/2201842839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330005040 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03409d9700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0330013510 tx=0x7f033000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0330009320 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0330013f00 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c108dc0 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0330014dc0 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.886+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c109250 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0330014420 con 0x7f033c0f51a0 2024-01-21T07:38:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c045ad0 con 0x7f033c0f51a0 2024-01-21T07:38:05.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f03397fa700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T07:38:05.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0330041110 con 0x7f033c0f51a0 2024-01-21T07:38:05.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f033bfff700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T07:38:05.890 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.887+0000 7f033bfff700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 0x7f032c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f03280096a0 tx=0x7f0328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T07:38:05.892 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:05.892+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03300095f0 con 0x7f033c0f51a0 2024-01-21T07:38:06.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.099+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f033c10a100 con 0x7f033c0f51a0 2024-01-21T07:38:06.100 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.100+0000 7f03397fa700 1 -- 172.21.15.115:0/2201842839 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f033001cb10 con 0x7f033c0f51a0 2024-01-21T07:38:06.101 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-01-21T07:38:06.101 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"962f5a9a-b82e-11ee-95b0-87774f69a715","modified":"2024-01-21T07:28:16.804927Z","created":"2024-01-21T07:28:16.804927Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi107","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T07:38:06.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 msgr2=0x7f032c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:06.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 0x7f032c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f03280096a0 tx=0x7f0328006b40).stop 2024-01-21T07:38:06.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 msgr2=0x7f033c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T07:38:06.102 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0330013510 tx=0x7f033000b040).stop 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 shutdown_connections 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:6800/1892560072,v1:172.21.15.107:6801/1892560072] conn(0x7f032c02ff60 0x7f032c032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 --2- 172.21.15.115:0/2201842839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f033c0f51a0 0x7f033c1086d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 >> 172.21.15.115:0/2201842839 conn(0x7f033c0f09a0 msgr2=0x7f033c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 shutdown_connections 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:2024-01-21T07:38:06.101+0000 7f03419db700 1 -- 172.21.15.115:0/2201842839 wait complete. 2024-01-21T07:38:06.103 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2024-01-21T07:38:06.315 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:05 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: cluster 2024-01-21T07:38:05.896542+0000 mgr.smithi107.adejht (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T07:38:06.514 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T07:38:06.515 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T07:38:06.516 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T07:38:06.539 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T07:38:06.558 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T07:38:06.559 INFO:tasks.cephadm.mon.smithi107:Stopping mon.smithi107... 2024-01-21T07:38:06.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107 2024-01-21T07:38:06.885 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:06 smithi107 systemd[1]: Stopping Ceph mon.smithi107 for 962f5a9a-b82e-11ee-95b0-87774f69a715... 2024-01-21T07:38:06.886 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:38:06.685+0000 7fdffd283700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi107 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T07:38:06.886 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Jan 21 07:38:06 smithi107 ceph-962f5a9a-b82e-11ee-95b0-87774f69a715-mon.smithi107[38761]: debug 2024-01-21T07:38:06.685+0000 7fdffd283700 -1 mon.smithi107@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T07:38:07.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi107.service' 2024-01-21T07:38:07.106 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T07:38:07.106 INFO:tasks.cephadm.mon.smithi107:Stopped mon.smithi107 2024-01-21T07:38:07.106 INFO:tasks.cephadm.mon.smithi115:Stopping mon.smithi115... 2024-01-21T07:38:07.106 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115 2024-01-21T07:38:07.140 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115.service: Unit ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115.service not loaded. 2024-01-21T07:38:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T07:38:07.142 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi115" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 5: 'sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115' 2024-01-21T07:38:07.143 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T07:38:07.145 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/ubuntu@smithi107.front.sepia.ceph.com/crash 2024-01-21T07:38:07.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/crash -- . 2024-01-21T07:38:07.185 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/ubuntu@smithi115.front.sepia.ceph.com/crash 2024-01-21T07:38:07.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/crash -- . 2024-01-21T07:38:07.228 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T07:38:07.228 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2024-01-21T07:38:07.229 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T07:38:07.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T07:38:07.259 INFO:teuthology.orchestra.run.smithi107.stderr:grep: /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T07:38:07.261 INFO:tasks.cephadm:Compressing logs... 2024-01-21T07:38:07.261 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T07:38:07.303 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T07:38:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-21T07:38:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T07:38:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mgr.smithi107.adejht.log 2024-01-21T07:38:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mon.smithi107.log 2024-01-21T07:38:07.329 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mgr.smithi107.adejht.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T07:38:07.329 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mon.smithi107.log: /var/log/ceph/cephadm.log: 90.8% -- replaced with /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T07:38:07.331 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-21T07:38:07.331 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T07:38:07.333 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T07:38:07.335 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-01-21T07:38:07.335 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m0.014s 2024-01-21T07:38:07.335 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m0.010s 2024-01-21T07:38:07.335 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.012s 2024-01-21T07:38:07.335 INFO:teuthology.orchestra.run.smithi107.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T07:38:07.406 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mgr.smithi107.adejht.log.gz 2024-01-21T07:38:07.471 INFO:teuthology.orchestra.run.smithi107.stderr: 90.7% -- replaced with /var/log/ceph/962f5a9a-b82e-11ee-95b0-87774f69a715/ceph-mon.smithi107.log.gz 2024-01-21T07:38:07.473 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-01-21T07:38:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.156s 2024-01-21T07:38:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m0.229s 2024-01-21T07:38:07.474 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.021s 2024-01-21T07:38:07.474 INFO:tasks.cephadm:Archiving logs... 2024-01-21T07:38:07.474 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/ubuntu@smithi107.front.sepia.ceph.com/log 2024-01-21T07:38:07.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T07:38:07.590 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/ubuntu@smithi115.front.sepia.ceph.com/log 2024-01-21T07:38:07.591 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T07:38:07.622 INFO:tasks.cephadm:Removing cluster... 2024-01-21T07:38:07.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 --force 2024-01-21T07:39:19.649 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 962f5a9a-b82e-11ee-95b0-87774f69a715 --force 2024-01-21T07:39:19.958 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T07:39:19.958 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T07:39:19.978 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T07:39:19.995 INFO:tasks.cephadm:Teardown complete 2024-01-21T07:39:19.996 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 5: 'sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115' 2024-01-21T07:39:20.214 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5127765b67244608847ae806d96c70be Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 5: 'sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115' 2024-01-21T07:39:20.218 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T07:39:20.228 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T07:39:20.238 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T07:39:20.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T07:39:20.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T07:39:20.336 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T07:39:20.345 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T07:39:20.345 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T07:39:20.348 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T07:39:20.362 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2024-01-21T07:39:20.364 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +659us[ +659us] +/- 86ms 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -861us[ -844us] +/- 68ms 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +464us[ +464us] +/- 83ms 2024-01-21T07:39:20.367 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:39:20.368 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T07:39:20.369 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-01-21T07:39:20.369 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +659us[ +662us] +/- 86ms 2024-01-21T07:39:20.369 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -853us[ -850us] +/- 68ms 2024-01-21T07:39:20.369 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +442us[ +442us] +/- 83ms 2024-01-21T07:39:20.369 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T07:39:20.370 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T07:39:20.395 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T07:39:20.396 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T07:39:20.406 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T07:39:20.469 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T07:39:20.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:39:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:39:20.561 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T07:39:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:39:20.591 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T07:39:20.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T07:39:20.609 INFO:teuthology.task.internal:Duration was 1229.277357 seconds 2024-01-21T07:39:20.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T07:39:20.620 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T07:39:20.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T07:39:20.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T07:39:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:39:20.676 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T07:39:21.013 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T07:39:21.013 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-01-21T07:39:21.013 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T07:39:21.036 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-01-21T07:39:21.037 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T07:39:21.058 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T07:39:21.059 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T07:39:21.079 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T07:39:21.148 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T07:39:21.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T07:39:21.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T07:39:21.239 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T07:39:21.249 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T07:39:21.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T07:39:21.276 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T07:39:21.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T07:39:21.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T07:39:21.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T07:39:21.343 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-01-21T07:39:21.370 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-01-21T07:39:21.423 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T07:39:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:39:21.440 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T07:39:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T07:39:21.480 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T07:39:21.490 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T07:39:21.490 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/smithi107 2024-01-21T07:39:21.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T07:39:21.532 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228/remote/smithi115 2024-01-21T07:39:21.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T07:39:21.572 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T07:39:21.573 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T07:39:21.575 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T07:39:21.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T07:39:21.634 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T07:39:21.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T07:39:21.649 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T07:39:21.650 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T07:39:21.652 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T07:39:21.667 INFO:teuthology.orchestra.run.smithi107.stdout: 525828 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 07:39 /home/ubuntu/cephtest 2024-01-21T07:39:21.681 INFO:teuthology.orchestra.run.smithi115.stdout: 525838 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 07:39 /home/ubuntu/cephtest 2024-01-21T07:39:21.682 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T07:39:21.693 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T07:39:21.722 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T07:39:21.749 DEBUG:teuthology.nuke:shortname: smithi107 2024-01-21T07:39:21.750 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T07:39:21.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:10:40.288251', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuDCNfoxzHXB8fNTMhK4TGKKfDzG5B+jegV+VPwiryb1FXjUuBdz+DvMbp9Lxjl8omzQHsAGJ9DWURq/P/Iob0='} 2024-01-21T07:39:21.775 DEBUG:teuthology.nuke:shortname: smithi115 2024-01-21T07:39:21.775 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T07:39:21.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 07:10:40.289985', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyDHvJhRVbhUnxp4R/WyGlaU29uLAD6NJJRgUKS/1Rrj8sF9tlF89hKMVyA6WhQ7wChR4oSjmz6o2egi5TjIpQ='} 2024-01-21T07:39:21.830 INFO:teuthology.orchestra.console.smithi107:Power off 2024-01-21T07:39:21.830 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T07:39:21.854 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-01-21T07:39:21.855 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:21.872 INFO:teuthology.orchestra.console.smithi115:Power off 2024-01-21T07:39:21.872 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T07:39:21.882 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-01-21T07:39:21.893 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-01-21T07:39:21.894 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:21.915 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-01-21T07:39:25.883 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:25.917 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:25.993 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-01-21T07:39:26.027 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-01-21T07:39:29.997 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:30.027 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:30.108 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-01-21T07:39:30.138 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-01-21T07:39:34.112 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:34.139 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T07:39:34.223 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-01-21T07:39:34.248 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-01-21T07:39:34.325 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-01-21T07:39:34.350 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-01-21T07:39:34.455 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1229.277357339859 failure_reason: 'Command failed on smithi115 with status 5: ''sudo systemctl stop ceph-962f5a9a-b82e-11ee-95b0-87774f69a715@mon.smithi115''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5127765b67244608847ae806d96c70be status: fail success: false 2024-01-21T07:39:34.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T07:39:34.541 INFO:teuthology.run:FAIL