2024-02-13T03:12:46.885 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-13T03:12:46.885 DEBUG:teuthology.run:Teuthology command: teuthology --name lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --verbose --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185 -- /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/orig.config.yaml 2024-02-13T03:12:46.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T03:12:47.021 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7557185' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-tracker-64343-pacific sha1: 84e714279aa684636ac645daf0a1a85c2094cfb3 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 84e714279aa684636ac645daf0a1a85c2094cfb3 targets: smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIs9gF0uJF7l4uk+bpLQcy2Tc+w2O03dLVx4pZs5u2B7ZrhFQQFpn0FjaFLEU+C1UxjL+WRs9ylCdLmwCO3LF4Y= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlhBeoPgnC/NZRGMfUuPLhAZfCfuZax//FmjalFq1XcVkxpCm+y4BCqsJXckUbkU6J/xEg/V5oKW6jdYQ0iP3c= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-12_21:56:27 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-13T03:12:47.021 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa; will attempt to use it 2024-02-13T03:12:47.022 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks 2024-02-13T03:12:47.022 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-13T03:12:47.023 INFO:teuthology.task.internal:Checking packages... 2024-02-13T03:12:47.046 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-13T03:12:47.047 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-13T03:12:47.047 INFO:teuthology.packaging:ref: None 2024-02-13T03:12:47.047 INFO:teuthology.packaging:tag: None 2024-02-13T03:12:47.047 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-13T03:12:47.047 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:12:47.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-13T03:12:47.309 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-13T03:12:47.311 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-13T03:12:47.317 INFO:teuthology.task.internal:no buildpackages task found 2024-02-13T03:12:47.317 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-13T03:12:47.323 INFO:teuthology.task.internal:Saving configuration 2024-02-13T03:12:47.342 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-13T03:12:47.347 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T03:12:47.374 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 03:07:41.733154', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIs9gF0uJF7l4uk+bpLQcy2Tc+w2O03dLVx4pZs5u2B7ZrhFQQFpn0FjaFLEU+C1UxjL+WRs9ylCdLmwCO3LF4Y='} 2024-02-13T03:12:47.395 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 03:07:41.731401', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlhBeoPgnC/NZRGMfUuPLhAZfCfuZax//FmjalFq1XcVkxpCm+y4BCqsJXckUbkU6J/xEg/V5oKW6jdYQ0iP3c='} 2024-02-13T03:12:47.395 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-13T03:12:47.401 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-13T03:12:47.401 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-13T03:12:47.401 INFO:teuthology.run_tasks:Running task console_log... 2024-02-13T03:12:47.471 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f915583ff70>, signals=[15]) 2024-02-13T03:12:47.471 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-13T03:12:47.478 INFO:teuthology.task.internal:Opening connections... 2024-02-13T03:12:47.478 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-02-13T03:12:47.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:12:47.556 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-02-13T03:12:47.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:12:47.629 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-13T03:12:47.636 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-02-13T03:12:47.657 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-02-13T03:12:47.657 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="8" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="8" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el8" 2024-02-13T03:12:47.717 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-13T03:12:47.718 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T03:12:47.718 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-02-13T03:12:47.742 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-02-13T03:12:47.764 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-02-13T03:12:47.764 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2024-02-13T03:12:47.826 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-13T03:12:47.827 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T03:12:47.827 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-02-13T03:12:47.850 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-13T03:12:47.858 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-13T03:12:47.863 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-13T03:12:47.863 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-02-13T03:12:47.866 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-02-13T03:12:47.887 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-13T03:12:47.893 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-13T03:12:47.894 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-02-13T03:12:47.925 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-02-13T03:12:47.959 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-13T03:12:48.015 INFO:teuthology.run_tasks:Running task kernel... 2024-02-13T03:12:48.030 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-13T03:12:48.030 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-13T03:12:48.031 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T03:12:48.031 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T03:12:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:12:48.050 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-02-13T03:12:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:12:48.052 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-02-13T03:12:48.110 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-489.el8.x86_64 2024-02-13T03:12:48.110 INFO:teuthology.task.kernel:Running kernel on smithi080: 4.18.0-489.el8.x86_64 2024-02-13T03:12:48.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2024-02-13T03:12:48.113 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-489.el8.x86_64 2024-02-13T03:12:48.113 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-489.el8.x86_64 2024-02-13T03:12:48.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-02-13T03:12:58.407 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 3.4 MB/s | 27 MB 00:07 2024-02-13T03:13:03.665 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 10 MB 00:00 2024-02-13T03:13:05.591 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 52 kB/s | 18 kB 00:00 2024-02-13T03:13:06.108 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-02-13T03:13:06.351 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-13T03:13:06.595 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - AppStream 1.7 MB/s | 27 MB 00:15 2024-02-13T03:13:07.077 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-02-13T03:13:10.793 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 448 kB/s | 24 kB 00:00 2024-02-13T03:13:11.819 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 10 MB 00:00 2024-02-13T03:13:13.723 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-13T03:13:13.724 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-13T03:13:13.745 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2024-02-13T03:13:13.843 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-13T03:13:13.845 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:Install 3 Packages 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 90 M 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 96 M 2024-02-13T03:13:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:13:14.404 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-02-13T03:13:14.520 INFO:teuthology.orchestra.run.smithi112.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 21 MB/s | 10 MB 00:00 2024-02-13T03:13:14.647 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-13T03:13:15.320 INFO:teuthology.orchestra.run.smithi112.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 28 MB/s | 36 MB 00:01 2024-02-13T03:13:15.737 INFO:teuthology.orchestra.run.smithi112.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 26 MB/s | 44 MB 00:01 2024-02-13T03:13:15.738 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:13:15.738 INFO:teuthology.orchestra.run.smithi112.stdout:Total 48 MB/s | 90 MB 00:01 2024-02-13T03:13:16.413 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:13:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:13:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:13:18.059 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:13:18.083 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:13:18.575 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 16 MB 00:03 2024-02-13T03:13:20.865 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:13:21.335 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:13:22.281 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 500 kB/s | 24 kB 00:00 2024-02-13T03:13:24.904 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:13:25.169 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-13T03:13:25.169 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-13T03:13:25.268 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:13:25.277 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:13:25.277 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:13:25.277 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout:Install 3 Packages 2024-02-13T03:13:25.278 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:13:25.279 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 90 M 2024-02-13T03:13:25.279 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 96 M 2024-02-13T03:13:25.279 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:13:25.915 INFO:teuthology.orchestra.run.smithi080.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 24 MB/s | 10 MB 00:00 2024-02-13T03:13:26.749 INFO:teuthology.orchestra.run.smithi080.stdout:(2/3): kernel-core-4.18.0-536.el8.x86_64.rpm 34 MB/s | 44 MB 00:01 2024-02-13T03:13:27.199 INFO:teuthology.orchestra.run.smithi080.stdout:(3/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2024-02-13T03:13:27.199 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:13:27.199 INFO:teuthology.orchestra.run.smithi080.stdout:Total 47 MB/s | 90 MB 00:01 2024-02-13T03:13:27.880 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:13:28.302 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:13:28.302 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:13:28.702 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:13:28.984 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:29.536 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:13:29.557 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:13:32.306 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:13:32.806 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:13:36.374 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:13:36.788 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:13:40.226 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:13:40.528 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:51.286 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:51.309 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:51.653 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:51.653 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:13:51.653 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:13:52.163 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:13:52.163 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:52.163 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:13:52.164 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-13T03:13:52.164 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-13T03:13:52.164 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:52.164 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:13:52.908 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T03:13:53.127 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T03:13:53.127 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T03:13:53.128 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-13T03:13:53.128 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T03:13:53.128 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T03:13:53.128 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-13T03:13:53.128 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T03:13:53.128 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T03:13:53.337 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T03:13:53.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T03:13:53.338 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.02 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 129.el8 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Group : System Environment/Base 2024-02-13T03:13:53.371 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 9265586 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Relocations : (not relocatable) 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : CentOS BuildSystem 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T03:13:53.372 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T03:13:53.373 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T03:13:53.373 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-02-13T03:13:53.373 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:13:53.373 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T03:13:53.374 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T03:13:53.374 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 4.18.0-536.el8.x86_64 2024-02-13T03:13:53.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T03:13:54.683 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-02-13T03:13:57.047 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-02-13T03:13:57.049 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-02-13T03:13:57.077 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.6NYm5FUhWd 2024-02-13T03:13:57.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6NYm5FUhWd 2024-02-13T03:13:57.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.6NYm5FUhWd 2024-02-13T03:13:57.292 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.6NYm5FUhWd is 6KB 2024-02-13T03:13:57.340 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.6NYm5FUhWd 2024-02-13T03:13:57.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T03:13:57.425 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-13T03:13:57.426 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-13T03:13:57.426 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-13T03:13:57.426 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-13T03:13:57.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-13T03:13:57.599 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-02-13T03:14:02.715 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:14:02.738 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:14:03.082 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:14:03.082 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-13T03:14:03.082 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-13T03:14:03.592 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:14:03.593 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:14:03.776 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T03:14:03.997 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T03:14:03.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T03:14:03.997 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-13T03:14:03.997 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T03:14:03.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T03:14:03.997 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-13T03:14:03.998 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T03:14:03.998 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T03:14:04.207 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-13T03:14:04.208 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-13T03:14:04.208 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.02 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 129.el8 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Group : System Environment/Base 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 9265586 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-13T03:14:04.242 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:Relocations : (not relocatable) 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : CentOS BuildSystem 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:14:04.243 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T03:14:04.244 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T03:14:04.244 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 4.18.0-536.el8.x86_64 2024-02-13T03:14:04.244 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T03:14:05.899 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-02-13T03:14:08.319 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-02-13T03:14:08.320 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-02-13T03:14:08.404 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.xEt9x4lFGN 2024-02-13T03:14:08.404 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xEt9x4lFGN 2024-02-13T03:14:08.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.xEt9x4lFGN 2024-02-13T03:14:08.617 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.xEt9x4lFGN is 6KB 2024-02-13T03:14:08.665 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.xEt9x4lFGN 2024-02-13T03:14:08.683 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T03:14:08.757 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-13T03:14:08.757 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-13T03:14:08.758 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-13T03:14:08.758 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-13T03:14:08.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-13T03:14:08.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-02-13T03:14:27.620 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-13T03:14:27.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:14:27.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:14:38.940 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-13T03:14:38.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:14:38.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:14:45.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-02-13T03:14:54.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:14:54.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:14:57.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-02-13T03:14:58.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-02-13T03:15:06.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:15:06.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:15:09.536 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-02-13T03:15:10.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:15:10.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:15:21.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:15:21.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:15:28.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-02-13T03:15:43.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:15:43.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:15:44.747 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-02-13T03:15:45.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:15:45.513 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-02-13T03:15:45.513 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-02-13T03:15:45.530 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-536.el8.x86_64 2024-02-13T03:15:45.530 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-13T03:15:45.530 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T03:15:45.531 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T03:15:46.531 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-13T03:15:46.532 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T03:15:46.617 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-02-13T03:15:46.630 DEBUG:teuthology.parallel:result is None 2024-02-13T03:15:53.392 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-02-13T03:15:53.647 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:15:53.647 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-02-13T03:15:53.647 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-02-13T03:15:53.664 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-536.el8.x86_64 2024-02-13T03:15:53.664 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-13T03:15:53.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T03:15:53.664 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T03:15:54.665 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-13T03:15:54.665 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T03:15:54.732 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-02-13T03:15:54.744 DEBUG:teuthology.parallel:result is None 2024-02-13T03:15:54.745 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-13T03:15:54.754 INFO:teuthology.task.internal:Creating test directory... 2024-02-13T03:15:54.754 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T03:15:54.756 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T03:15:54.778 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-13T03:15:54.786 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-13T03:15:54.800 INFO:teuthology.task.internal:Creating archive directory... 2024-02-13T03:15:54.800 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T03:15:54.816 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T03:15:54.859 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-13T03:15:54.870 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-13T03:15:54.871 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T03:15:54.895 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T03:15:54.930 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T03:15:54.935 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T03:15:54.941 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T03:15:54.947 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T03:15:54.949 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-13T03:15:54.957 INFO:teuthology.task.internal:Configuring sudo... 2024-02-13T03:15:54.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T03:15:54.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T03:15:55.012 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-13T03:15:55.022 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-13T03:15:55.023 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T03:15:55.030 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T03:15:55.072 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T03:15:55.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T03:15:55.189 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T03:15:55.272 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T03:15:55.319 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:15:55.319 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T03:15:55.389 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T03:15:55.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T03:15:55.491 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T03:15:55.577 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T03:15:55.622 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:15:55.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T03:15:55.691 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-02-13T03:15:55.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-02-13T03:15:55.763 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T03:15:55.785 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T03:15:56.266 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-13T03:15:56.276 INFO:teuthology.task.internal:Starting timer... 2024-02-13T03:15:56.276 INFO:teuthology.run_tasks:Running task pcp... 2024-02-13T03:15:56.293 INFO:teuthology.run_tasks:Running task selinux... 2024-02-13T03:15:56.303 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-13T03:15:56.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-02-13T03:15:56.390 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-02-13T03:15:56.392 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-02-13T03:15:56.478 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-02-13T03:15:56.480 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-13T03:15:56.480 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-02-13T03:15:56.541 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-02-13T03:15:56.541 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-02-13T03:15:56.564 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-02-13T03:15:56.564 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2024-02-13T03:15:56.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T03:15:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:15:56.592 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T03:15:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:15:56.632 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-13T03:15:56.632 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-02-13T03:15:56.667 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-02-13T03:15:56.699 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-13T03:15:56.710 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-13T03:15:56.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-13T03:15:56.727 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi080.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-13T03:21:43.417 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2024-02-13T03:21:43.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:21:43.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:21:43.496 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-02-13T03:21:43.563 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-02-13T03:21:43.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:21:43.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T03:21:43.637 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-02-13T03:21:43.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-02-13T03:21:43.709 INFO:teuthology.run_tasks:Running task clock... 2024-02-13T03:21:43.724 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-13T03:21:43.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T03:21:43.725 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T03:21:43.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T03:21:43.727 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T03:21:43.759 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T03:21:43.773 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T03:21:43.780 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T03:21:43.794 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T03:21:43.838 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-02-13T03:21:43.850 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-02-13T03:21:43.857 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-02-13T03:21:43.864 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T03:21:43.867 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-02-13T03:21:43.877 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T03:21:43.882 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T03:21:43.895 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T03:21:43.941 INFO:teuthology.orchestra.run.smithi080.stderr:bash: ntpq: command not found 2024-02-13T03:21:43.944 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T03:21:43.944 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-02-13T03:21:43.945 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.945 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.945 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.945 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.960 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:21:43.965 INFO:teuthology.run_tasks:Running task pexec... 2024-02-13T03:21:43.975 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-13T03:21:43.975 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi080.front.sepia.ceph.com 2024-02-13T03:21:43.976 DEBUG:teuthology.orchestra.run.smithi080:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T03:21:43.976 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi112.front.sepia.ceph.com 2024-02-13T03:21:43.976 DEBUG:teuthology.orchestra.run.smithi112:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T03:21:44.767 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:25 ago on Tue 13 Feb 2024 03:20:19 AM UTC. 2024-02-13T03:21:44.788 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:25 ago on Tue 13 Feb 2024 03:20:19 AM UTC. 2024-02-13T03:21:46.710 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout:Resetting modules: 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout: container-tools 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:46.711 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:21:46.747 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:Resetting modules: 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:46.959 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:21:47.008 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:21:47.793 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:28 ago on Tue 13 Feb 2024 03:20:19 AM UTC. 2024-02-13T03:21:47.859 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:28 ago on Tue 13 Feb 2024 03:20:19 AM UTC. 2024-02-13T03:21:49.766 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout:Installing group/module packages: 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-13T03:21:49.801 INFO:teuthology.orchestra.run.smithi080.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:Installing weak dependencies: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:Installing module profiles: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: container-tools/common 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:Enabling module streams: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: container-tools rhel8 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:21:49.802 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:21:49.803 INFO:teuthology.orchestra.run.smithi080.stdout:Install 13 Packages 2024-02-13T03:21:49.803 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:21:49.803 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 27 M 2024-02-13T03:21:49.803 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 91 M 2024-02-13T03:21:49.803 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:21:49.811 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:21:49.846 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:49.846 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:21:49.846 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:49.846 INFO:teuthology.orchestra.run.smithi112.stdout:Installing group/module packages: 2024-02-13T03:21:49.846 INFO:teuthology.orchestra.run.smithi112.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-02-13T03:21:49.847 INFO:teuthology.orchestra.run.smithi112.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout:Installing module profiles: 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools/common 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout:Enabling module streams: 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout: container-tools rhel8 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-02-13T03:21:49.848 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:49.849 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 27 M 2024-02-13T03:21:49.849 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 91 M 2024-02-13T03:21:49.849 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:21:50.476 INFO:teuthology.orchestra.run.smithi080.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.2 MB/s | 759 kB 00:00 2024-02-13T03:21:50.508 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.2 MB/s | 759 kB 00:00 2024-02-13T03:21:50.543 INFO:teuthology.orchestra.run.smithi080.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.4 MB/s | 1.0 MB 00:00 2024-02-13T03:21:50.585 INFO:teuthology.orchestra.run.smithi080.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 349 kB/s | 38 kB 00:00 2024-02-13T03:21:50.625 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 325 kB/s | 38 kB 00:00 2024-02-13T03:21:50.675 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.3 MB/s | 1.0 MB 00:00 2024-02-13T03:21:50.677 INFO:teuthology.orchestra.run.smithi080.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.8 MB/s | 248 kB 00:00 2024-02-13T03:21:50.750 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.9 MB/s | 248 kB 00:00 2024-02-13T03:21:50.867 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.5 MB/s | 179 kB 00:00 2024-02-13T03:21:50.968 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2024-02-13T03:21:51.076 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 867 kB/s | 94 kB 00:00 2024-02-13T03:21:51.352 INFO:teuthology.orchestra.run.smithi080.stdout:(5/13): netavark-1.9.0-1.module_el8+804+f131391 5.2 MB/s | 4.0 MB 00:00 2024-02-13T03:21:51.461 INFO:teuthology.orchestra.run.smithi080.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 228 kB/s | 25 kB 00:00 2024-02-13T03:21:51.578 INFO:teuthology.orchestra.run.smithi080.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 805 kB/s | 94 kB 00:00 2024-02-13T03:21:51.643 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 4.1 MB/s | 4.0 MB 00:00 2024-02-13T03:21:51.729 INFO:teuthology.orchestra.run.smithi080.stdout:(8/13): python3-podman-4.8.2-1.module_el8+804+f 171 kB/s | 179 kB 00:01 2024-02-13T03:21:52.060 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 4.4 MB/s | 9.7 MB 00:02 2024-02-13T03:21:52.177 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 452 kB/s | 53 kB 00:00 2024-02-13T03:21:52.277 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): skopeo-1.14.0-2.module_el8+804+f131391 7.3 MB/s | 8.8 MB 00:01 2024-02-13T03:21:52.302 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 327 kB/s | 41 kB 00:00 2024-02-13T03:21:52.362 INFO:teuthology.orchestra.run.smithi080.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 3.8 MB/s | 9.7 MB 00:02 2024-02-13T03:21:52.373 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): toolbox-0.0.99.4-8.module_el8+804+f131 3.4 MB/s | 2.5 MB 00:00 2024-02-13T03:21:52.373 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:21:52.373 INFO:teuthology.orchestra.run.smithi112.stdout:Total 11 MB/s | 27 MB 00:02 2024-02-13T03:21:52.479 INFO:teuthology.orchestra.run.smithi080.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 452 kB/s | 53 kB 00:00 2024-02-13T03:21:52.596 INFO:teuthology.orchestra.run.smithi080.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 351 kB/s | 41 kB 00:00 2024-02-13T03:21:52.644 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:21:52.665 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:21:52.665 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:21:52.940 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:21:52.943 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:21:53.355 INFO:teuthology.orchestra.run.smithi080.stdout:(12/13): toolbox-0.0.99.4-8.module_el8+804+f131 1.5 MB/s | 2.5 MB 00:01 2024-02-13T03:21:53.547 INFO:teuthology.orchestra.run.smithi080.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 4.5 MB/s | 8.8 MB 00:01 2024-02-13T03:21:53.547 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:21:53.547 INFO:teuthology.orchestra.run.smithi080.stdout:Total 7.3 MB/s | 27 MB 00:03 2024-02-13T03:21:53.628 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:21:53.817 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:21:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-13T03:21:53.838 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:21:53.838 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:21:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-13T03:21:54.099 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:21:54.102 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:21:54.192 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-13T03:21:54.425 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T03:21:54.792 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:21:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-13T03:21:55.045 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-13T03:21:55.223 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-13T03:21:55.439 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-13T03:21:55.700 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T03:21:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T03:21:56.194 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-13T03:21:56.296 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-13T03:21:56.397 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-13T03:21:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-13T03:21:57.054 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-13T03:21:57.279 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T03:21:57.511 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-13T03:21:57.748 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-13T03:21:57.977 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-13T03:21:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T03:21:58.223 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-13T03:21:58.395 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-13T03:21:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-13T03:21:58.850 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-13T03:21:58.851 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-13T03:21:58.851 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T03:21:58.851 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-13T03:21:59.340 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-13T03:21:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:21:59.486 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:21:59.586 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-13T03:21:59.743 DEBUG:teuthology.parallel:result is None 2024-02-13T03:21:59.830 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-13T03:22:00.238 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-13T03:22:00.239 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-13T03:22:00.722 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:22:00.723 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:22:01.317 DEBUG:teuthology.parallel:result is None 2024-02-13T03:22:01.317 INFO:teuthology.run_tasks:Running task pexec... 2024-02-13T03:22:01.326 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-13T03:22:01.326 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi080.front.sepia.ceph.com 2024-02-13T03:22:01.327 DEBUG:teuthology.orchestra.run.smithi080:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T03:22:01.327 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi112.front.sepia.ceph.com 2024-02-13T03:22:01.327 DEBUG:teuthology.orchestra.run.smithi112:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T03:22:01.482 DEBUG:teuthology.parallel:result is None 2024-02-13T03:22:01.547 DEBUG:teuthology.parallel:result is None 2024-02-13T03:22:01.547 INFO:teuthology.run_tasks:Running task install... 2024-02-13T03:22:01.556 DEBUG:teuthology.task.install:project ceph 2024-02-13T03:22:01.556 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-13T03:22:01.556 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-13T03:22:01.556 INFO:teuthology.task.install:Using flavor: default 2024-02-13T03:22:01.565 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-13T03:22:01.565 INFO:teuthology.task.install:extra packages: [] 2024-02-13T03:22:01.565 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-13T03:22:01.566 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:22:01.566 INFO:teuthology.packaging:ref: None 2024-02-13T03:22:01.566 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:22:01.566 INFO:teuthology.packaging:branch: None 2024-02-13T03:22:01.566 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:22:01.979 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-13T03:22:01.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-13T03:22:01.982 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-13T03:22:01.982 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:22:01.982 INFO:teuthology.packaging:ref: None 2024-02-13T03:22:01.982 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:22:01.982 INFO:teuthology.packaging:branch: None 2024-02-13T03:22:01.983 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:22:01.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-13T03:22:02.129 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-13T03:22:02.129 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-13T03:22:02.132 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-13T03:22:02.133 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-13T03:22:02.171 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-13T03:22:02.172 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:22:02.172 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-13T03:22:02.181 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-13T03:22:02.181 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:22:02.181 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-13T03:22:02.199 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-13T03:22:02.199 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:22:02.200 INFO:teuthology.packaging:ref: None 2024-02-13T03:22:02.200 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:22:02.200 INFO:teuthology.packaging:branch: None 2024-02-13T03:22:02.200 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:22:02.200 DEBUG:teuthology.orchestra.run.smithi080:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-13T03:22:02.214 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-13T03:22:02.214 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:22:02.214 INFO:teuthology.packaging:ref: None 2024-02-13T03:22:02.214 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:22:02.214 INFO:teuthology.packaging:branch: None 2024-02-13T03:22:02.215 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:22:02.215 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-13T03:22:02.267 DEBUG:teuthology.orchestra.run.smithi080:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-13T03:22:02.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-13T03:22:02.344 DEBUG:teuthology.orchestra.run.smithi080:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-13T03:22:02.367 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-13T03:22:02.410 INFO:teuthology.orchestra.run.smithi080.stdout:check_obsoletes = 1 2024-02-13T03:22:02.412 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum clean all 2024-02-13T03:22:02.435 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-02-13T03:22:02.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-02-13T03:22:02.683 INFO:teuthology.orchestra.run.smithi080.stdout:76 files removed 2024-02-13T03:22:02.711 INFO:teuthology.orchestra.run.smithi112.stdout:76 files removed 2024-02-13T03:22:02.713 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-radosgw 2024-02-13T03:22:02.738 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2024-02-13T03:22:03.212 INFO:teuthology.orchestra.run.smithi080.stdout:ceph packages for x86_64 792 kB/s | 74 kB 00:00 2024-02-13T03:22:03.213 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-02-13T03:22:03.330 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-02-13T03:22:03.353 INFO:teuthology.orchestra.run.smithi080.stdout:ceph noarch packages 148 kB/s | 15 kB 00:00 2024-02-13T03:22:03.431 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-13T03:22:03.469 INFO:teuthology.orchestra.run.smithi080.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-02-13T03:22:07.694 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - AppStream 6.4 MB/s | 27 MB 00:04 2024-02-13T03:22:07.815 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 6.2 MB/s | 27 MB 00:04 2024-02-13T03:22:12.995 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-02-13T03:22:13.291 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-02-13T03:22:14.912 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras 54 kB/s | 18 kB 00:00 2024-02-13T03:22:15.116 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 97 kB/s | 18 kB 00:00 2024-02-13T03:22:15.562 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-02-13T03:22:15.794 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-02-13T03:22:16.697 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - PowerTools 4.6 MB/s | 4.8 MB 00:01 2024-02-13T03:22:16.760 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - PowerTools 5.4 MB/s | 4.8 MB 00:00 2024-02-13T03:22:17.748 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-13T03:22:17.895 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-02-13T03:22:18.374 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 30 MB/s | 16 MB 00:00 2024-02-13T03:22:18.929 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-02-13T03:22:22.290 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 417 kB/s | 24 kB 00:00 2024-02-13T03:22:22.762 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 256 kB/s | 24 kB 00:00 2024-02-13T03:22:25.783 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:22:25.786 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrading: 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-13T03:22:25.787 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-13T03:22:25.788 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:Install 25 Packages 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrade 2 Packages 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 59 M 2024-02-13T03:22:25.789 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:22:26.272 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:22:26.274 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:22:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-13T03:22:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-13T03:22:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout:Install 25 Packages 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-02-13T03:22:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:22:26.278 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 59 M 2024-02-13T03:22:26.278 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:22:26.358 INFO:teuthology.orchestra.run.smithi080.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2024-02-13T03:22:26.375 INFO:teuthology.orchestra.run.smithi080.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-02-13T03:22:26.408 INFO:teuthology.orchestra.run.smithi080.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-13T03:22:26.442 INFO:teuthology.orchestra.run.smithi080.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-13T03:22:26.542 INFO:teuthology.orchestra.run.smithi080.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-13T03:22:26.567 INFO:teuthology.orchestra.run.smithi080.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-13T03:22:26.618 INFO:teuthology.orchestra.run.smithi080.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.5 MB/s | 80 kB 00:00 2024-02-13T03:22:26.652 INFO:teuthology.orchestra.run.smithi080.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2024-02-13T03:22:26.685 INFO:teuthology.orchestra.run.smithi080.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-13T03:22:26.819 INFO:teuthology.orchestra.run.smithi080.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-02-13T03:22:26.852 INFO:teuthology.orchestra.run.smithi080.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 368 kB 00:00 2024-02-13T03:22:26.877 INFO:teuthology.orchestra.run.smithi080.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 114 kB 00:00 2024-02-13T03:22:27.007 INFO:teuthology.orchestra.run.smithi112.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-02-13T03:22:27.145 INFO:teuthology.orchestra.run.smithi080.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-13T03:22:27.266 INFO:teuthology.orchestra.run.smithi112.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 95 kB/s | 25 kB 00:00 2024-02-13T03:22:27.332 INFO:teuthology.orchestra.run.smithi112.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 801 kB 00:00 2024-02-13T03:22:27.365 INFO:teuthology.orchestra.run.smithi112.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-13T03:22:27.466 INFO:teuthology.orchestra.run.smithi112.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-13T03:22:27.491 INFO:teuthology.orchestra.run.smithi112.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-13T03:22:27.516 INFO:teuthology.orchestra.run.smithi112.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-13T03:22:27.541 INFO:teuthology.orchestra.run.smithi112.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-13T03:22:27.584 INFO:teuthology.orchestra.run.smithi112.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2024-02-13T03:22:27.617 INFO:teuthology.orchestra.run.smithi112.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-13T03:22:27.642 INFO:teuthology.orchestra.run.smithi112.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-13T03:22:27.649 INFO:teuthology.orchestra.run.smithi080.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 182 kB/s | 145 kB 00:00 2024-02-13T03:22:27.801 INFO:teuthology.orchestra.run.smithi112.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 11 MB/s | 13 MB 00:01 2024-02-13T03:22:27.900 INFO:teuthology.orchestra.run.smithi080.stdout:(15/27): librdkafka-1.6.1-1.el8.x86_64.rpm 662 kB/s | 677 kB 00:01 2024-02-13T03:22:28.043 INFO:teuthology.orchestra.run.smithi080.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 289 kB/s | 259 kB 00:00 2024-02-13T03:22:28.068 INFO:teuthology.orchestra.run.smithi112.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 17 MB/s | 24 MB 00:01 2024-02-13T03:22:28.071 INFO:teuthology.orchestra.run.smithi080.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 104 kB/s | 44 kB 00:00 2024-02-13T03:22:28.130 INFO:teuthology.orchestra.run.smithi080.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 183 kB/s | 42 kB 00:00 2024-02-13T03:22:28.155 INFO:teuthology.orchestra.run.smithi080.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-02-13T03:22:28.160 INFO:teuthology.orchestra.run.smithi112.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 281 kB/s | 145 kB 00:00 2024-02-13T03:22:28.180 INFO:teuthology.orchestra.run.smithi080.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 740 kB/s | 80 kB 00:00 2024-02-13T03:22:28.269 INFO:teuthology.orchestra.run.smithi112.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 406 kB/s | 44 kB 00:00 2024-02-13T03:22:28.327 INFO:teuthology.orchestra.run.smithi112.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 723 kB/s | 42 kB 00:00 2024-02-13T03:22:28.361 INFO:teuthology.orchestra.run.smithi112.stdout:(17/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-02-13T03:22:28.394 INFO:teuthology.orchestra.run.smithi112.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 2.4 MB/s | 80 kB 00:00 2024-02-13T03:22:28.489 INFO:teuthology.orchestra.run.smithi080.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 349 kB/s | 116 kB 00:00 2024-02-13T03:22:28.531 INFO:teuthology.orchestra.run.smithi080.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 516 kB/s | 181 kB 00:00 2024-02-13T03:22:28.573 INFO:teuthology.orchestra.run.smithi080.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 704 kB/s | 59 kB 00:00 2024-02-13T03:22:28.598 INFO:teuthology.orchestra.run.smithi080.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 692 kB/s | 306 kB 00:00 2024-02-13T03:22:28.603 INFO:teuthology.orchestra.run.smithi112.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 844 kB/s | 677 kB 00:00 2024-02-13T03:22:28.623 INFO:teuthology.orchestra.run.smithi080.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 815 kB/s | 75 kB 00:00 2024-02-13T03:22:28.628 INFO:teuthology.orchestra.run.smithi112.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 464 kB/s | 259 kB 00:00 2024-02-13T03:22:28.721 INFO:teuthology.orchestra.run.smithi112.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 357 kB/s | 116 kB 00:00 2024-02-13T03:22:28.730 INFO:teuthology.orchestra.run.smithi080.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2024-02-13T03:22:28.799 INFO:teuthology.orchestra.run.smithi080.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2024-02-13T03:22:28.804 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:22:28.804 INFO:teuthology.orchestra.run.smithi080.stdout:Total 20 MB/s | 59 MB 00:03 2024-02-13T03:22:28.812 INFO:teuthology.orchestra.run.smithi112.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 646 kB/s | 59 kB 00:00 2024-02-13T03:22:28.870 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:22:28.904 INFO:teuthology.orchestra.run.smithi112.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 819 kB/s | 75 kB 00:00 2024-02-13T03:22:28.917 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:22:28.918 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:22:28.988 INFO:teuthology.orchestra.run.smithi112.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2024-02-13T03:22:29.014 INFO:teuthology.orchestra.run.smithi112.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 470 kB/s | 181 kB 00:00 2024-02-13T03:22:29.038 INFO:teuthology.orchestra.run.smithi112.stdout:(26/27): gperftools-libs-2.7-9.el8.x86_64.rpm 704 kB/s | 306 kB 00:00 2024-02-13T03:22:29.114 INFO:teuthology.orchestra.run.smithi112.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.0 MB 00:00 2024-02-13T03:22:29.118 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:22:29.118 INFO:teuthology.orchestra.run.smithi112.stdout:Total 21 MB/s | 59 MB 00:02 2024-02-13T03:22:29.189 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:22:29.237 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:22:29.237 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:22:29.253 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:22:29.256 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:22:29.595 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:22:29.598 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:22:30.364 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:22:30.567 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-13T03:22:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:22:30.804 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-13T03:22:30.964 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-13T03:22:30.999 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-13T03:22:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-13T03:22:31.336 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-13T03:22:31.398 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-13T03:22:31.624 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:22:31.715 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:22:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-13T03:22:31.951 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-13T03:22:31.994 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:22:32.090 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:22:32.107 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:22:32.169 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:22:32.409 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-13T03:22:32.673 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-13T03:22:32.836 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:22:32.886 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:22:32.967 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-13T03:22:33.138 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-13T03:22:33.341 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:22:33.602 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:22:33.670 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-13T03:22:33.802 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:22:33.838 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-13T03:22:33.900 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:22:33.999 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-13T03:22:34.065 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-13T03:22:34.067 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-13T03:22:34.227 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-13T03:22:34.255 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-13T03:22:34.316 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-13T03:22:34.466 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-13T03:22:34.474 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-13T03:22:34.632 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-13T03:22:34.644 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-13T03:22:34.693 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-13T03:22:34.819 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-13T03:22:34.871 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:22:34.893 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-13T03:22:35.045 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:22:35.303 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:22:35.482 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:22:35.507 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-13T03:22:35.600 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-13T03:22:35.693 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-13T03:22:35.774 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-13T03:22:35.835 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-13T03:22:35.972 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-13T03:22:36.075 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-13T03:22:36.187 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-13T03:22:36.231 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-13T03:22:36.310 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-13T03:22:36.368 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-13T03:22:36.427 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-13T03:22:36.557 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-13T03:22:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-13T03:22:36.756 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-13T03:22:36.826 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-13T03:22:36.939 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-13T03:22:37.000 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-13T03:22:37.123 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-13T03:22:37.235 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-13T03:22:39.680 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:39.800 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:39.915 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:40.611 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:40.703 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-13T03:22:40.898 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-13T03:22:40.985 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-13T03:22:40.999 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-13T03:22:41.091 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-13T03:22:41.155 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:22:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:23:00.380 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:23:00.380 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /sys 2024-02-13T03:23:00.380 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /proc 2024-02-13T03:23:00.380 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /run 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /mnt 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /var/tmp 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /home 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /tmp 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2024-02-13T03:23:00.381 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-02-13T03:23:00.387 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-02-13T03:23:00.388 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-02-13T03:23:00.388 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-02-13T03:23:00.388 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2024-02-13T03:23:00.388 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-13T03:23:01.710 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-13T03:23:01.776 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-13T03:23:01.776 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-13T03:23:01.809 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-13T03:23:01.810 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-13T03:23:01.987 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-13T03:23:01.987 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:02.050 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-13T03:23:02.051 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:23:02.694 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-13T03:23:02.695 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-13T03:23:02.696 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-13T03:23:02.742 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-13T03:23:02.743 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-13T03:23:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-13T03:23:03.584 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout:Upgraded: 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: leveldb-1.22-1.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-13T03:23:03.585 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.586 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.587 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.587 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:03.587 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:03.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-13T03:23:03.657 INFO:teuthology.orchestra.run.smithi112.stdout: leveldb-1.22-1.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-13T03:23:03.658 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:03.659 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:03.659 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:03.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-test 2024-02-13T03:23:03.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2024-02-13T03:23:04.654 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:42 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:04.719 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:42 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:06.163 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:06.164 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-02-13T03:23:06.165 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:06.166 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 53 M 2024-02-13T03:23:06.166 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 229 M 2024-02-13T03:23:06.166 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:06.200 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-13T03:23:06.201 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-02-13T03:23:06.202 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:06.203 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 53 M 2024-02-13T03:23:06.203 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 229 M 2024-02-13T03:23:06.203 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:06.350 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 169 kB 00:00 2024-02-13T03:23:06.424 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-02-13T03:23:06.968 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 72 MB/s | 52 MB 00:00 2024-02-13T03:23:06.993 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 271 kB/s | 203 kB 00:00 2024-02-13T03:23:07.018 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 282 kB/s | 188 kB 00:00 2024-02-13T03:23:07.099 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2024-02-13T03:23:07.124 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 266 kB/s | 203 kB 00:00 2024-02-13T03:23:07.126 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 653 kB/s | 71 kB 00:00 2024-02-13T03:23:07.149 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 259 kB/s | 188 kB 00:00 2024-02-13T03:23:07.266 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 605 kB/s | 71 kB 00:00 2024-02-13T03:23:07.493 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 615 kB/s | 323 kB 00:00 2024-02-13T03:23:07.494 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:07.495 INFO:teuthology.orchestra.run.smithi112.stdout:Total 41 MB/s | 53 MB 00:01 2024-02-13T03:23:07.537 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:23:07.558 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:23:07.558 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:23:07.624 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 615 kB/s | 323 kB 00:00 2024-02-13T03:23:07.626 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:07.626 INFO:teuthology.orchestra.run.smithi080.stdout:Total 36 MB/s | 53 MB 00:01 2024-02-13T03:23:07.667 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:23:07.687 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:23:07.687 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:23:07.876 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:23:07.879 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:23:08.003 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:23:08.006 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:23:08.393 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:23:08.492 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:23:08.658 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-13T03:23:08.744 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-13T03:23:08.840 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-13T03:23:08.965 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-13T03:23:09.023 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T03:23:09.098 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T03:23:09.141 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T03:23:09.245 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-13T03:23:09.275 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-13T03:23:09.442 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-13T03:23:09.447 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-13T03:23:09.591 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-13T03:23:14.284 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-13T03:23:14.426 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-13T03:23:14.495 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-13T03:23:14.774 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-13T03:23:14.863 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-13T03:23:15.156 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-13T03:23:15.156 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-13T03:23:15.156 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-13T03:23:15.156 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-13T03:23:15.156 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-13T03:23:15.157 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-13T03:23:15.410 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-13T03:23:15.410 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:15.410 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:15.410 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-13T03:23:15.410 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-13T03:23:15.411 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-13T03:23:15.411 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:15.411 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-13T03:23:15.590 INFO:teuthology.orchestra.run.smithi080.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-13T03:23:15.591 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:15.591 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:15.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2024-02-13T03:23:15.826 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph 2024-02-13T03:23:16.443 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:54 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:16.583 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:54 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:17.950 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:17.953 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:17.953 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:23:17.953 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:17.953 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-13T03:23:17.954 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-13T03:23:17.955 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout:Install 32 Packages 2024-02-13T03:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 31 M 2024-02-13T03:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 111 M 2024-02-13T03:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:18.091 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-13T03:23:18.095 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-13T03:23:18.096 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:18.097 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout:Install 32 Packages 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 31 M 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 111 M 2024-02-13T03:23:18.098 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:18.270 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 128 kB/s | 6.3 kB 00:00 2024-02-13T03:23:18.320 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-13T03:23:18.395 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-13T03:23:18.420 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-13T03:23:18.496 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-13T03:23:18.543 INFO:teuthology.orchestra.run.smithi080.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 130 kB/s | 6.3 kB 00:00 2024-02-13T03:23:18.593 INFO:teuthology.orchestra.run.smithi080.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-13T03:23:18.644 INFO:teuthology.orchestra.run.smithi080.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-02-13T03:23:18.669 INFO:teuthology.orchestra.run.smithi080.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-13T03:23:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-02-13T03:23:18.752 INFO:teuthology.orchestra.run.smithi080.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-02-13T03:23:18.929 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 89 kB/s | 39 kB 00:00 2024-02-13T03:23:18.955 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 294 kB/s | 157 kB 00:00 2024-02-13T03:23:18.995 INFO:teuthology.orchestra.run.smithi080.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-02-13T03:23:19.122 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 297 kB/s | 57 kB 00:00 2024-02-13T03:23:19.164 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 238 kB/s | 103 kB 00:00 2024-02-13T03:23:19.187 INFO:teuthology.orchestra.run.smithi080.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 89 kB/s | 39 kB 00:00 2024-02-13T03:23:19.212 INFO:teuthology.orchestra.run.smithi080.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 289 kB/s | 157 kB 00:00 2024-02-13T03:23:19.222 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.4 MB/s | 249 kB 00:00 2024-02-13T03:23:19.256 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2024-02-13T03:23:19.281 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 5.3 MB/s | 641 kB 00:00 2024-02-13T03:23:19.379 INFO:teuthology.orchestra.run.smithi080.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 297 kB/s | 57 kB 00:00 2024-02-13T03:23:19.429 INFO:teuthology.orchestra.run.smithi080.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 238 kB/s | 103 kB 00:00 2024-02-13T03:23:19.432 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 249 kB/s | 44 kB 00:00 2024-02-13T03:23:19.463 INFO:teuthology.orchestra.run.smithi080.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.9 MB/s | 249 kB 00:00 2024-02-13T03:23:19.497 INFO:teuthology.orchestra.run.smithi080.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2024-02-13T03:23:19.507 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 823 kB/s | 185 kB 00:00 2024-02-13T03:23:19.530 INFO:teuthology.orchestra.run.smithi080.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 6.2 MB/s | 641 kB 00:00 2024-02-13T03:23:19.565 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-02-13T03:23:19.624 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2024-02-13T03:23:19.649 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-werkzeug-0.12.2-4.el8.noarch.r 659 kB/s | 457 kB 00:00 2024-02-13T03:23:19.674 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2024-02-13T03:23:19.700 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 232 kB/s | 18 kB 00:00 2024-02-13T03:23:19.733 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-02-13T03:23:19.793 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2024-02-13T03:23:19.817 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-02-13T03:23:19.842 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): python3-logutils-0.3.5-11.el8.noarch.r 253 kB/s | 49 kB 00:00 2024-02-13T03:23:19.868 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 247 kB/s | 19 kB 00:00 2024-02-13T03:23:19.893 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-13T03:23:19.897 INFO:teuthology.orchestra.run.smithi080.stdout:(14/32): python3-werkzeug-0.12.2-4.el8.noarch.r 668 kB/s | 457 kB 00:00 2024-02-13T03:23:19.918 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-02-13T03:23:19.922 INFO:teuthology.orchestra.run.smithi080.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 102 kB/s | 44 kB 00:00 2024-02-13T03:23:19.944 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 363 kB/s | 27 kB 00:00 2024-02-13T03:23:19.969 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-13T03:23:20.019 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-02-13T03:23:20.044 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-13T03:23:20.078 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-13T03:23:20.079 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:20.080 INFO:teuthology.orchestra.run.smithi112.stdout:Total 15 MB/s | 31 MB 00:02 2024-02-13T03:23:20.148 INFO:teuthology.orchestra.run.smithi080.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 300 kB/s | 185 kB 00:00 2024-02-13T03:23:20.161 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:23:20.199 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:23:20.200 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:23:20.223 INFO:teuthology.orchestra.run.smithi080.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 530 kB/s | 173 kB 00:00 2024-02-13T03:23:20.248 INFO:teuthology.orchestra.run.smithi080.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-02-13T03:23:20.332 INFO:teuthology.orchestra.run.smithi080.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-13T03:23:20.382 INFO:teuthology.orchestra.run.smithi080.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 365 kB/s | 49 kB 00:00 2024-02-13T03:23:20.474 INFO:teuthology.orchestra.run.smithi080.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 414 kB/s | 59 kB 00:00 2024-02-13T03:23:20.529 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:23:20.532 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:23:20.591 INFO:teuthology.orchestra.run.smithi080.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 134 kB/s | 16 kB 00:00 2024-02-13T03:23:20.633 INFO:teuthology.orchestra.run.smithi080.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-02-13T03:23:20.700 INFO:teuthology.orchestra.run.smithi080.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-02-13T03:23:20.734 INFO:teuthology.orchestra.run.smithi080.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-02-13T03:23:20.800 INFO:teuthology.orchestra.run.smithi080.stdout:(26/32): python3-cherrypy-18.4.0-1.el8.noarch.r 438 kB/s | 384 kB 00:00 2024-02-13T03:23:20.826 INFO:teuthology.orchestra.run.smithi080.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-02-13T03:23:20.851 INFO:teuthology.orchestra.run.smithi080.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-02-13T03:23:20.976 INFO:teuthology.orchestra.run.smithi080.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 683 kB/s | 85 kB 00:00 2024-02-13T03:23:21.018 INFO:teuthology.orchestra.run.smithi080.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-02-13T03:23:21.076 INFO:teuthology.orchestra.run.smithi080.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-02-13T03:23:21.159 INFO:teuthology.orchestra.run.smithi080.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 712 kB/s | 255 kB 00:00 2024-02-13T03:23:21.164 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:21.164 INFO:teuthology.orchestra.run.smithi080.stdout:Total 10 MB/s | 31 MB 00:03 2024-02-13T03:23:21.242 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:23:21.280 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:23:21.281 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:23:21.598 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:23:21.601 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:23:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:23:21.884 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-13T03:23:22.095 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-13T03:23:22.330 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-13T03:23:22.552 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-13T03:23:22.703 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:23:22.725 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-13T03:23:22.918 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-13T03:23:22.925 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-13T03:23:23.082 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-13T03:23:23.104 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-13T03:23:23.251 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-13T03:23:23.388 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-13T03:23:23.476 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-13T03:23:23.602 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-13T03:23:23.681 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-13T03:23:23.784 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-13T03:23:23.927 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-13T03:23:24.005 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-13T03:23:24.191 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-13T03:23:24.209 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-13T03:23:24.385 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-13T03:23:24.559 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-13T03:23:24.575 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-13T03:23:24.798 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-13T03:23:24.830 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-13T03:23:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-13T03:23:25.052 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-13T03:23:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-13T03:23:25.292 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-13T03:23:25.388 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T03:23:25.672 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-13T03:23:25.675 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-13T03:23:25.901 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-13T03:23:25.932 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-13T03:23:26.081 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-13T03:23:26.120 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-13T03:23:26.315 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-13T03:23:26.370 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-13T03:23:26.547 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T03:23:26.578 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-13T03:23:26.849 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-13T03:23:26.863 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-13T03:23:27.059 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-13T03:23:27.126 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-13T03:23:27.248 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-13T03:23:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-13T03:23:27.480 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-13T03:23:27.726 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-13T03:23:27.734 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-13T03:23:27.963 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-13T03:23:28.151 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-13T03:23:28.180 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-13T03:23:28.370 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-13T03:23:28.512 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-13T03:23:28.825 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-13T03:23:29.241 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-13T03:23:29.437 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-13T03:23:29.917 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-13T03:23:30.144 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-13T03:23:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-13T03:23:30.962 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-13T03:23:31.020 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-13T03:23:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-13T03:23:31.437 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-13T03:23:31.454 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-13T03:23:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-13T03:23:31.619 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-13T03:23:31.870 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-13T03:23:31.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-13T03:23:31.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-13T03:23:32.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-13T03:23:32.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-13T03:23:32.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-13T03:23:32.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-13T03:23:32.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-13T03:23:32.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-13T03:23:32.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-13T03:23:32.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-13T03:23:32.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-13T03:23:32.092 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-13T03:23:32.286 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-13T03:23:32.304 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-13T03:23:32.461 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-13T03:23:32.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-13T03:23:32.793 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-13T03:23:32.794 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-13T03:23:32.795 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:32.796 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-13T03:23:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-13T03:23:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-13T03:23:32.869 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-13T03:23:32.870 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-13T03:23:32.871 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-13T03:23:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-13T03:23:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-13T03:23:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-13T03:23:32.872 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-13T03:23:33.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:33.693 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend-2.6-1.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-13T03:23:33.694 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:33.695 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:33.798 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:11 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:33.987 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-base 2024-02-13T03:23:34.741 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:12 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:35.274 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:23:35.320 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:35.321 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:23:35.321 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:35.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2024-02-13T03:23:36.167 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:14 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:36.233 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:23:36.279 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:36.280 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:23:36.281 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:36.392 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install cephadm 2024-02-13T03:23:37.134 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:15 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:37.677 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:37.677 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:23:37.678 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:37.679 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 71 k 2024-02-13T03:23:37.679 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 290 k 2024-02-13T03:23:37.679 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:37.732 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-02-13T03:23:37.733 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:37.733 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-02-13T03:23:37.733 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:23:37.739 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:23:37.739 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:23:37.741 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:23:37.744 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:23:37.787 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:23:38.261 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:38.473 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:38.650 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:38.651 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:23:38.652 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:38.652 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 71 k 2024-02-13T03:23:38.652 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 290 k 2024-02-13T03:23:38.652 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:38.689 INFO:teuthology.orchestra.run.smithi080.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 71 kB 00:00 2024-02-13T03:23:38.690 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:38.690 INFO:teuthology.orchestra.run.smithi080.stdout:Total 1.8 MB/s | 71 kB 00:00 2024-02-13T03:23:38.690 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:23:38.697 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:23:38.697 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:23:38.698 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:38.698 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:23:38.702 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:23:38.743 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:23:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:39.089 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:39.089 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-13T03:23:39.090 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:39.090 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:39.256 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:39.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2024-02-13T03:23:39.431 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:39.659 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:40.057 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:40.096 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:18 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:40.319 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-immutable-object-cache 2024-02-13T03:23:41.074 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:19 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:41.607 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:23:41.608 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:41.609 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 163 k 2024-02-13T03:23:41.609 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 587 k 2024-02-13T03:23:41.609 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:41.978 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 442 kB/s | 163 kB 00:00 2024-02-13T03:23:41.978 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:41.979 INFO:teuthology.orchestra.run.smithi112.stdout:Total 440 kB/s | 163 kB 00:00 2024-02-13T03:23:41.979 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:23:41.990 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:23:41.990 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:23:42.040 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:23:42.043 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:23:42.587 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:42.587 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:23:42.588 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:42.589 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 163 k 2024-02-13T03:23:42.589 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 587 k 2024-02-13T03:23:42.589 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:42.636 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2024-02-13T03:23:42.636 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:42.637 INFO:teuthology.orchestra.run.smithi080.stdout:Total 3.3 MB/s | 163 kB 00:00 2024-02-13T03:23:42.637 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:23:42.647 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:23:42.647 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:23:42.694 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:23:42.697 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:23:42.815 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:23:42.833 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:23:43.000 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.018 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.444 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.484 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.794 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:43.794 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:43.794 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:43.794 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:43.794 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:43.903 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:44.039 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2024-02-13T03:23:44.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr 2024-02-13T03:23:44.803 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:22 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:44.849 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:22 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:46.259 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:23:46.302 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:23:46.306 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:46.307 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:23:46.307 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:46.349 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:46.350 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:23:46.350 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:46.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2024-02-13T03:23:46.457 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-dashboard 2024-02-13T03:23:47.160 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:25 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:47.218 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:25 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:48.654 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:48.655 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:23:48.656 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-02-13T03:23:48.656 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:48.656 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.0 M 2024-02-13T03:23:48.656 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 15 M 2024-02-13T03:23:48.656 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:48.744 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:48.744 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:48.745 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:23:48.746 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-02-13T03:23:48.746 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:48.746 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.0 M 2024-02-13T03:23:48.746 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 15 M 2024-02-13T03:23:48.746 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:48.898 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 434 kB/s | 24 kB 00:00 2024-02-13T03:23:48.923 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 111 kB/s | 8.9 kB 00:00 2024-02-13T03:23:48.981 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-02-13T03:23:49.022 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 530 kB/s | 24 kB 00:00 2024-02-13T03:23:49.032 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 317 kB/s | 43 kB 00:00 2024-02-13T03:23:49.039 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 144 kB/s | 8.9 kB 00:00 2024-02-13T03:23:49.065 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 236 kB/s | 34 kB 00:00 2024-02-13T03:23:49.106 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2024-02-13T03:23:49.191 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 940 kB/s | 196 kB 00:00 2024-02-13T03:23:49.193 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:49.193 INFO:teuthology.orchestra.run.smithi080.stdout:Total 6.6 MB/s | 3.0 MB 00:00 2024-02-13T03:23:49.198 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2024-02-13T03:23:49.202 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:23:49.211 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:23:49.211 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:23:49.371 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:23:49.374 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:23:49.390 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 96 kB/s | 34 kB 00:00 2024-02-13T03:23:49.649 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 362 kB/s | 196 kB 00:00 2024-02-13T03:23:49.651 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:23:49.652 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2024-02-13T03:23:49.661 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:23:49.666 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:23:49.671 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:23:49.671 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:23:49.825 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:23:49.828 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:23:49.888 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-13T03:23:50.121 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:23:50.192 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-13T03:23:50.339 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-13T03:23:50.401 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-13T03:23:50.530 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-13T03:23:50.626 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-13T03:23:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-13T03:23:50.940 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-13T03:23:51.496 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-13T03:23:51.782 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:23:51.923 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-13T03:23:52.063 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-13T03:23:52.191 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-13T03:23:52.451 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-13T03:23:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-13T03:23:52.452 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:52.452 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:23:52.462 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:23:52.462 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-13T03:23:52.462 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-13T03:23:52.462 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-13T03:23:52.462 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-13T03:23:52.463 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-13T03:23:52.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-13T03:23:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:52.845 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:23:52.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-13T03:23:53.326 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:31 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:53.729 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:31 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:23:54.820 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:23:54.821 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-02-13T03:23:54.821 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:23:54.821 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:23:54.822 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-02-13T03:23:54.823 INFO:teuthology.orchestra.run.smithi080.stdout:Install 8 Packages 2024-02-13T03:23:54.823 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:23:54.823 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 37 M 2024-02-13T03:23:54.823 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 210 M 2024-02-13T03:23:54.823 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:23:55.078 INFO:teuthology.orchestra.run.smithi080.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 49 MB/s | 8.6 MB 00:00 2024-02-13T03:23:55.248 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-13T03:23:55.249 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================================ 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout:Install 8 Packages 2024-02-13T03:23:55.250 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:23:55.251 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 37 M 2024-02-13T03:23:55.251 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 210 M 2024-02-13T03:23:55.251 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:23:55.620 INFO:teuthology.orchestra.run.smithi112.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 61 MB/s | 8.6 MB 00:00 2024-02-13T03:23:57.536 INFO:teuthology.orchestra.run.smithi080.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.8 MB/s | 4.9 MB 00:02 2024-02-13T03:23:57.678 INFO:teuthology.orchestra.run.smithi080.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.4 MB/s | 3.7 MB 00:02 2024-02-13T03:23:57.703 INFO:teuthology.orchestra.run.smithi080.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2024-02-13T03:23:57.796 INFO:teuthology.orchestra.run.smithi080.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 6.9 MB/s | 645 kB 00:00 2024-02-13T03:23:57.837 INFO:teuthology.orchestra.run.smithi080.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.1 MB/s | 172 kB 00:00 2024-02-13T03:23:58.071 INFO:teuthology.orchestra.run.smithi080.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 1.5 MB/s | 4.7 MB 00:03 2024-02-13T03:23:59.313 INFO:teuthology.orchestra.run.smithi112.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 1.2 MB/s | 4.7 MB 00:03 2024-02-13T03:23:59.580 INFO:teuthology.orchestra.run.smithi112.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 965 kB/s | 3.7 MB 00:03 2024-02-13T03:23:59.605 INFO:teuthology.orchestra.run.smithi112.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 772 kB/s | 225 kB 00:00 2024-02-13T03:23:59.697 INFO:teuthology.orchestra.run.smithi112.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 6.9 MB/s | 645 kB 00:00 2024-02-13T03:23:59.731 INFO:teuthology.orchestra.run.smithi112.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2024-02-13T03:24:00.123 INFO:teuthology.orchestra.run.smithi112.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.0 MB/s | 4.9 MB 00:04 2024-02-13T03:24:06.698 INFO:teuthology.orchestra.run.smithi080.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.6 MB/s | 14 MB 00:09 2024-02-13T03:24:06.698 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:06.699 INFO:teuthology.orchestra.run.smithi080.stdout:Total 3.1 MB/s | 37 MB 00:11 2024-02-13T03:24:06.930 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:06.948 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:06.948 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:07.279 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:07.282 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:07.884 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:08.020 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T03:24:08.172 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T03:24:08.295 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T03:24:08.887 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T03:24:09.103 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:09.729 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:09.919 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T03:24:10.595 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T03:24:10.930 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T03:24:11.136 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T03:24:12.364 INFO:teuthology.orchestra.run.smithi112.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.1 MB/s | 14 MB 00:12 2024-02-13T03:24:12.365 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:12.365 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.2 MB/s | 37 MB 00:17 2024-02-13T03:24:12.597 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:12.615 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:12.615 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:12.891 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T03:24:12.944 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:13.600 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:13.722 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T03:24:13.845 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-13T03:24:13.989 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T03:24:14.193 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-13T03:24:14.377 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-13T03:24:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-13T03:24:14.680 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-13T03:24:14.681 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-13T03:24:14.755 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:15.118 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-13T03:24:15.118 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:15.118 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:15.118 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:15.119 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:15.226 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-rook 2024-02-13T03:24:15.371 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T03:24:15.976 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:53 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:16.228 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-13T03:24:16.575 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T03:24:16.830 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T03:24:17.474 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-13T03:24:17.476 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout:Install 13 Packages 2024-02-13T03:24:17.477 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:17.478 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 1.9 M 2024-02-13T03:24:17.478 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 24 M 2024-02-13T03:24:17.478 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:24:18.537 INFO:teuthology.orchestra.run.smithi080.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 727 kB/s | 27 kB 00:00 2024-02-13T03:24:18.584 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-13T03:24:18.770 INFO:teuthology.orchestra.run.smithi080.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 74 kB/s | 20 kB 00:00 2024-02-13T03:24:18.795 INFO:teuthology.orchestra.run.smithi080.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 90 kB/s | 27 kB 00:00 2024-02-13T03:24:18.871 INFO:teuthology.orchestra.run.smithi080.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-02-13T03:24:18.904 INFO:teuthology.orchestra.run.smithi080.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2024-02-13T03:24:18.963 INFO:teuthology.orchestra.run.smithi080.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 571 kB/s | 110 kB 00:00 2024-02-13T03:24:19.013 INFO:teuthology.orchestra.run.smithi080.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 265 kB/s | 126 kB 00:00 2024-02-13T03:24:19.439 INFO:teuthology.orchestra.run.smithi080.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 62 kB/s | 33 kB 00:00 2024-02-13T03:24:19.464 INFO:teuthology.orchestra.run.smithi080.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 33 kB/s | 16 kB 00:00 2024-02-13T03:24:19.489 INFO:teuthology.orchestra.run.smithi080.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 171 kB/s | 81 kB 00:00 2024-02-13T03:24:19.590 INFO:teuthology.orchestra.run.smithi080.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 613 kB/s | 61 kB 00:00 2024-02-13T03:24:19.614 INFO:teuthology.orchestra.run.smithi080.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 427 kB/s | 64 kB 00:00 2024-02-13T03:24:19.765 INFO:teuthology.orchestra.run.smithi080.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.6 MB/s | 1.2 MB 00:00 2024-02-13T03:24:19.768 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:19.768 INFO:teuthology.orchestra.run.smithi080.stdout:Total 854 kB/s | 1.9 MB 00:02 2024-02-13T03:24:19.827 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:19.844 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:19.844 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:19.896 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-13T03:24:19.938 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:19.941 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-13T03:24:20.413 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-13T03:24:20.413 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-13T03:24:20.413 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-13T03:24:20.413 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-13T03:24:20.413 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-13T03:24:20.414 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-13T03:24:20.414 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-13T03:24:20.414 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-13T03:24:20.440 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:20.678 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-13T03:24:20.899 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-13T03:24:21.007 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-13T03:24:21.008 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-13T03:24:21.008 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-13T03:24:21.008 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:21.008 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:21.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2024-02-13T03:24:21.161 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-13T03:24:21.390 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-13T03:24:21.597 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-13T03:24:21.820 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-13T03:24:21.865 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:59 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:22.086 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-13T03:24:22.320 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-13T03:24:23.333 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-13T03:24:23.357 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:23.359 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:24:23.359 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-13T03:24:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:23.361 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.9 M 2024-02-13T03:24:23.362 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2024-02-13T03:24:23.362 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:24:23.589 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-13T03:24:23.811 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-13T03:24:23.811 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 587 kB/s | 27 kB 00:00 2024-02-13T03:24:24.023 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-13T03:24:24.044 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 71 kB/s | 20 kB 00:00 2024-02-13T03:24:24.069 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 87 kB/s | 27 kB 00:00 2024-02-13T03:24:24.145 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-02-13T03:24:24.174 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-13T03:24:24.178 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2024-02-13T03:24:24.237 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 571 kB/s | 110 kB 00:00 2024-02-13T03:24:24.263 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 280 kB/s | 126 kB 00:00 2024-02-13T03:24:24.304 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 262 kB/s | 33 kB 00:00 2024-02-13T03:24:24.329 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2024-02-13T03:24:24.388 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 647 kB/s | 81 kB 00:00 2024-02-13T03:24:24.413 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 767 kB/s | 64 kB 00:00 2024-02-13T03:24:24.430 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.4 MB/s | 61 kB 00:00 2024-02-13T03:24:24.490 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.3 MB/s | 1.2 MB 00:00 2024-02-13T03:24:24.490 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:24.490 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-02-13T03:24:24.545 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:24.560 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:24.560 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:24.598 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-13T03:24:24.598 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-13T03:24:24.598 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-13T03:24:24.598 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-13T03:24:24.599 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-13T03:24:24.646 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:24.649 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:25.069 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-13T03:24:25.069 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:25.069 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:25.070 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:25.179 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:25.189 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-cephadm 2024-02-13T03:24:25.429 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-13T03:24:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-13T03:24:25.947 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:03 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:25.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-13T03:24:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-13T03:24:26.469 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-13T03:24:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-13T03:24:27.028 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-13T03:24:27.243 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-13T03:24:27.460 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:27.461 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:27.462 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-02-13T03:24:27.463 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:27.463 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 5.8 M 2024-02-13T03:24:27.463 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 24 M 2024-02-13T03:24:27.463 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:24:27.884 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2024-02-13T03:24:28.234 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 61 kB/s | 21 kB 00:00 2024-02-13T03:24:28.272 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-13T03:24:28.451 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 885 kB/s | 538 kB 00:00 2024-02-13T03:24:28.535 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-13T03:24:28.601 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 969 kB/s | 355 kB 00:00 2024-02-13T03:24:28.793 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-13T03:24:28.865 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 118 kB/s | 49 kB 00:00 2024-02-13T03:24:28.977 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-13T03:24:29.153 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-13T03:24:29.449 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-13T03:24:29.450 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-13T03:24:29.922 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:29.923 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:30.002 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.2 MB/s | 4.8 MB 00:02 2024-02-13T03:24:30.004 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:30.004 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.3 MB/s | 5.8 MB 00:02 2024-02-13T03:24:30.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2024-02-13T03:24:30.057 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:30.067 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:30.067 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:30.171 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:30.174 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:30.492 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:30.755 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-13T03:24:30.791 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:08 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:31.067 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-13T03:24:31.987 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-13T03:24:32.300 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-13T03:24:32.301 INFO:teuthology.orchestra.run.smithi112.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:32.302 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 5.8 M 2024-02-13T03:24:32.303 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 24 M 2024-02-13T03:24:32.303 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:24:32.351 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-13T03:24:32.556 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.3 MB/s | 89 kB 00:00 2024-02-13T03:24:32.602 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-13T03:24:32.731 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 121 kB/s | 21 kB 00:00 2024-02-13T03:24:32.797 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:24:32.940 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.7 MB/s | 355 kB 00:00 2024-02-13T03:24:33.007 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 734 kB/s | 49 kB 00:00 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-13T03:24:33.077 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-13T03:24:33.132 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 876 kB/s | 538 kB 00:00 2024-02-13T03:24:33.474 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:33.475 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:33.624 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-fuse 2024-02-13T03:24:34.369 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:12 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:34.574 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.3 MB/s | 4.8 MB 00:02 2024-02-13T03:24:34.575 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:34.575 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.6 MB/s | 5.8 MB 00:02 2024-02-13T03:24:34.629 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:34.639 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:34.639 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:34.741 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:34.744 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:35.040 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:35.326 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-13T03:24:35.588 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-13T03:24:35.856 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-02-13T03:24:35.857 INFO:teuthology.orchestra.run.smithi080.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout:Install 2 Packages 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 968 k 2024-02-13T03:24:35.858 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 3.2 M 2024-02-13T03:24:35.859 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:24:35.952 INFO:teuthology.orchestra.run.smithi080.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-13T03:24:35.977 INFO:teuthology.orchestra.run.smithi080.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2024-02-13T03:24:35.978 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:35.978 INFO:teuthology.orchestra.run.smithi080.stdout:Total 7.9 MB/s | 968 kB 00:00 2024-02-13T03:24:35.987 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:36.008 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:36.008 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:36.093 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:36.096 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:36.274 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:36.522 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-13T03:24:36.532 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-13T03:24:36.730 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-13T03:24:36.881 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-13T03:24:37.110 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-13T03:24:37.110 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-13T03:24:37.115 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-13T03:24:37.327 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:24:37.492 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-13T03:24:37.493 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:37.493 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:37.493 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-13T03:24:37.493 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:37.493 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-13T03:24:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-13T03:24:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-13T03:24:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-13T03:24:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-13T03:24:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-13T03:24:37.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados-devel 2024-02-13T03:24:38.037 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-13T03:24:38.037 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:38.038 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:38.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2024-02-13T03:24:38.458 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:16 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:38.957 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:16 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:39.975 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:39.975 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:24:39.976 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:39.977 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 131 k 2024-02-13T03:24:39.977 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 439 k 2024-02-13T03:24:39.977 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:24:40.031 INFO:teuthology.orchestra.run.smithi080.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-02-13T03:24:40.032 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:40.032 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-02-13T03:24:40.032 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:40.037 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:40.037 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:40.086 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:40.089 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:40.460 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:40.461 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:40.461 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:40.461 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:40.461 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:24:40.461 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 968 k 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 3.2 M 2024-02-13T03:24:40.462 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:24:40.550 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 18 MB/s | 885 kB 00:00 2024-02-13T03:24:40.589 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:40.599 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 856 kB/s | 83 kB 00:00 2024-02-13T03:24:40.600 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:40.600 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.8 MB/s | 968 kB 00:00 2024-02-13T03:24:40.609 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:40.630 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:40.630 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:40.717 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:40.720 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:41.202 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-13T03:24:41.295 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-13T03:24:41.594 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-13T03:24:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-13T03:24:41.973 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:41.973 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:41.974 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:41.974 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-13T03:24:41.974 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:41.974 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:42.176 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs2 2024-02-13T03:24:42.197 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-13T03:24:42.198 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:42.198 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:42.198 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-13T03:24:42.198 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:42.198 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:42.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2024-02-13T03:24:42.939 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:20 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:43.188 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:21 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:44.414 INFO:teuthology.orchestra.run.smithi080.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:44.460 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:44.461 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:24:44.461 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:44.577 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs-devel 2024-02-13T03:24:44.697 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-13T03:24:44.699 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 131 k 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 439 k 2024-02-13T03:24:44.700 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:24:44.745 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2024-02-13T03:24:44.745 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:44.745 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.8 MB/s | 131 kB 00:00 2024-02-13T03:24:44.745 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:44.750 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:44.750 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:44.800 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:44.803 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:45.167 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:45.325 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:23 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:45.833 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-13T03:24:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:45.833 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:46.839 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:46.840 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:46.840 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:46.840 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:46.841 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 24 k 2024-02-13T03:24:46.842 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 94 k 2024-02-13T03:24:46.842 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:24:47.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2024-02-13T03:24:47.354 INFO:teuthology.orchestra.run.smithi080.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 47 kB/s | 24 kB 00:00 2024-02-13T03:24:47.355 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:47.355 INFO:teuthology.orchestra.run.smithi080.stdout:Total 47 kB/s | 24 kB 00:00 2024-02-13T03:24:47.355 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:24:47.360 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:24:47.360 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:24:47.362 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:24:47.366 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:24:47.417 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:24:47.602 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:47.869 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:48.100 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:26 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:48.221 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:24:48.221 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-13T03:24:48.222 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:24:48.222 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:48.338 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados2 2024-02-13T03:24:49.092 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:27 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:49.574 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:49.620 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:49.621 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:24:49.621 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:49.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2024-02-13T03:24:50.464 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:28 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:50.562 INFO:teuthology.orchestra.run.smithi080.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:50.609 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:50.610 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:24:50.610 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:50.718 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librbd1 2024-02-13T03:24:51.467 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:29 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:51.980 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:24:51.981 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:51.982 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 24 k 2024-02-13T03:24:51.982 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 94 k 2024-02-13T03:24:51.982 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:24:52.024 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 573 kB/s | 24 kB 00:00 2024-02-13T03:24:52.024 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:24:52.025 INFO:teuthology.orchestra.run.smithi112.stdout:Total 557 kB/s | 24 kB 00:00 2024-02-13T03:24:52.025 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:24:52.028 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:24:52.028 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:24:52.031 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:24:52.033 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:24:52.080 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:24:52.283 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:52.550 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:52.942 INFO:teuthology.orchestra.run.smithi080.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:52.987 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:52.988 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:24:52.988 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:53.094 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rados 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:24:53.563 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:53.841 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:31 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:53.936 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2024-02-13T03:24:54.684 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:32 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:55.311 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:55.357 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:55.358 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:24:55.358 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:55.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rgw 2024-02-13T03:24:56.148 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:56.193 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:56.194 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:24:56.195 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:56.215 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:34 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:56.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2024-02-13T03:24:57.084 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:35 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:57.672 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:57.718 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:24:57.719 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:24:57.719 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:24:57.821 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-cephfs 2024-02-13T03:24:58.533 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:24:58.575 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:36 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:24:58.579 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:24:58.580 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:24:58.580 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:24:58.685 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2024-02-13T03:24:59.438 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:37 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:00.027 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:00.073 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:25:00.074 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:25:00.074 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:25:00.183 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rbd 2024-02-13T03:25:00.902 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:00.933 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:38 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:00.948 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:00.949 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:25:00.949 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:01.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2024-02-13T03:25:01.805 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:39 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:02.405 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:02.451 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:25:02.452 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-02-13T03:25:02.453 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:25:02.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-fuse 2024-02-13T03:25:03.280 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:03.325 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:03.326 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:25:03.327 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:03.327 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:41 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:03.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2024-02-13T03:25:04.183 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:42 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-13T03:25:04.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 89 k 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 298 k 2024-02-13T03:25:04.853 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:25:04.947 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 945 kB/s | 89 kB 00:00 2024-02-13T03:25:04.948 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:04.948 INFO:teuthology.orchestra.run.smithi080.stdout:Total 933 kB/s | 89 kB 00:00 2024-02-13T03:25:04.949 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:25:04.958 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:25:04.958 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:25:05.007 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:25:05.010 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:25:05.114 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:25:05.306 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:05.586 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:05.649 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:05.694 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:05.695 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:25:05.695 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:05.809 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2024-02-13T03:25:05.970 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:05.971 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:05.971 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:25:05.971 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:05.971 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:05.971 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:25:06.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-mirror 2024-02-13T03:25:06.557 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:44 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:06.940 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:44 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:08.023 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-13T03:25:08.069 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:08.070 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-02-13T03:25:08.070 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:08.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2024-02-13T03:25:08.450 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:25:08.451 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:08.452 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:25:08.452 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:08.452 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.7 M 2024-02-13T03:25:08.452 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 18 M 2024-02-13T03:25:08.452 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:25:08.536 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2024-02-13T03:25:08.537 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:08.537 INFO:teuthology.orchestra.run.smithi080.stdout:Total 43 MB/s | 3.7 MB 00:00 2024-02-13T03:25:08.538 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:25:08.549 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:25:08.549 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:25:08.612 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:25:08.615 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:25:08.935 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:46 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:09.080 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:25:09.306 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:09.731 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:10.126 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:25:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:10.126 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:10.127 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:25:10.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-nbd 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-13T03:25:10.431 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 89 k 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 298 k 2024-02-13T03:25:10.432 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:25:10.483 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-02-13T03:25:10.484 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:10.485 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-02-13T03:25:10.485 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:25:10.494 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:25:10.494 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:25:10.544 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:25:10.547 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:25:10.657 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:25:10.847 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:11.123 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:11.123 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:49 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:11.508 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:11.508 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:11.508 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:25:11.508 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:11.509 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:11.509 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:11.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2024-02-13T03:25:12.488 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:50 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:12.626 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-02-13T03:25:12.626 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-02-13T03:25:12.627 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:12.628 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 173 k 2024-02-13T03:25:12.628 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 649 k 2024-02-13T03:25:12.628 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-02-13T03:25:12.702 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 173 kB 00:00 2024-02-13T03:25:12.703 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:12.703 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.2 MB/s | 173 kB 00:00 2024-02-13T03:25:12.703 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-02-13T03:25:12.714 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-02-13T03:25:12.714 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-02-13T03:25:12.764 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-02-13T03:25:12.767 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-02-13T03:25:12.881 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-02-13T03:25:13.096 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:13.386 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-02-13T03:25:13.770 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-02-13T03:25:13.999 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:13.999 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:13.999 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:25:14.000 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:14.001 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.7 M 2024-02-13T03:25:14.001 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 18 M 2024-02-13T03:25:14.001 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:25:14.067 DEBUG:teuthology.parallel:result is None 2024-02-13T03:25:14.077 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2024-02-13T03:25:14.077 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:14.078 INFO:teuthology.orchestra.run.smithi112.stdout:Total 48 MB/s | 3.7 MB 00:00 2024-02-13T03:25:14.078 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:25:14.090 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:25:14.090 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:25:14.154 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:25:14.157 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:25:14.622 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:25:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:15.246 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:15.631 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:15.829 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2024-02-13T03:25:16.592 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:54 ago on Tue 13 Feb 2024 03:22:22 AM UTC. 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-02-13T03:25:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:18.104 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 173 k 2024-02-13T03:25:18.105 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 649 k 2024-02-13T03:25:18.105 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-02-13T03:25:18.155 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 173 kB 00:00 2024-02-13T03:25:18.155 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-02-13T03:25:18.155 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.3 MB/s | 173 kB 00:00 2024-02-13T03:25:18.156 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-02-13T03:25:18.166 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-02-13T03:25:18.166 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-02-13T03:25:18.215 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-02-13T03:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-02-13T03:25:18.331 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-02-13T03:25:18.563 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:18.852 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:19.209 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-13T03:25:19.210 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:19.210 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-02-13T03:25:19.210 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-13T03:25:19.210 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:25:19.210 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-02-13T03:25:19.437 DEBUG:teuthology.parallel:result is None 2024-02-13T03:25:19.438 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:25:19.438 INFO:teuthology.packaging:ref: None 2024-02-13T03:25:19.438 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:25:19.438 INFO:teuthology.packaging:branch: None 2024-02-13T03:25:19.438 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:25:19.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-13T03:25:19.656 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-13T03:25:19.683 INFO:teuthology.orchestra.run.smithi080.stdout:16.2.4-0.el8 2024-02-13T03:25:19.684 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-13T03:25:19.684 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-13T03:25:19.686 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-13T03:25:19.686 INFO:teuthology.packaging:ref: None 2024-02-13T03:25:19.686 INFO:teuthology.packaging:tag: v16.2.4 2024-02-13T03:25:19.686 INFO:teuthology.packaging:branch: None 2024-02-13T03:25:19.687 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-13T03:25:19.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-13T03:25:19.830 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-13T03:25:19.852 INFO:teuthology.orchestra.run.smithi112.stdout:16.2.4-0.el8 2024-02-13T03:25:19.853 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-13T03:25:19.853 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-13T03:25:19.854 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-13T03:25:19.854 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:25:19.854 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-13T03:25:19.888 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:25:19.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-13T03:25:19.920 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-13T03:25:19.920 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:25:19.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2024-02-13T03:25:19.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-13T03:25:20.033 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:25:20.033 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-02-13T03:25:20.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-13T03:25:20.126 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-13T03:25:20.126 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:25:20.127 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-13T03:25:20.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-13T03:25:20.229 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:25:20.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-13T03:25:20.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-13T03:25:20.322 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-13T03:25:20.322 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:25:20.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/stdin-killer 2024-02-13T03:25:20.353 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-13T03:25:20.423 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:25:20.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-02-13T03:25:20.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-13T03:25:20.516 INFO:teuthology.run_tasks:Running task print... 2024-02-13T03:25:20.526 INFO:teuthology.task.print:**** done install task... 2024-02-13T03:25:20.526 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-13T03:25:20.619 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-13T03:25:20.619 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-13T03:25:20.619 INFO:tasks.cephadm:Cluster fsid is 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:20.619 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-13T03:25:20.619 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-13T03:25:20.620 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi080': '172.21.15.80', 'mon.smithi112': '172.21.15.112'} 2024-02-13T03:25:20.620 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-13T03:25:20.620 DEBUG:teuthology.orchestra.run.smithi080:> sudo hostname $(hostname -s) 2024-02-13T03:25:20.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-02-13T03:25:20.679 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-13T03:25:20.679 DEBUG:teuthology.orchestra.run.smithi080:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:20.907 INFO:teuthology.orchestra.run.smithi080.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 13 03:25 /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:20.907 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:21.108 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 13 03:25 /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:21.109 DEBUG:teuthology.orchestra.run.smithi080:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:21.131 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T03:25:21.160 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph 2024-02-13T03:25:21.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-02-13T03:25:21.230 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /etc/ceph 2024-02-13T03:25:21.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-02-13T03:25:21.302 INFO:tasks.cephadm:Writing seed config... 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-13T03:25:21.303 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-13T03:25:21.304 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-13T03:25:21.305 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-13T03:25:21.306 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-13T03:25:21.306 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:25:21.306 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-13T03:25:21.338 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 84391b2a-ca1f-11ee-95b9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-13T03:25:21.338 DEBUG:teuthology.orchestra.run.smithi080:mon.smithi080> sudo journalctl -f -n 0 -u ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service 2024-02-13T03:25:21.381 INFO:tasks.cephadm:Bootstrapping... 2024-02-13T03:25:21.382 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.80 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-13T03:25:21.413 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:-- Logs begin at Tue 2024-02-13 03:15:29 UTC. -- 2024-02-13T03:25:21.676 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-13T03:25:22.696 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.8.3 2024-02-13T03:25:22.704 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying podman|docker is present... 2024-02-13T03:25:22.705 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-13T03:25:22.728 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.8.3 2024-02-13T03:25:22.739 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying lvm2 is present... 2024-02-13T03:25:22.739 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying time synchronization is in place... 2024-02-13T03:25:22.739 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled chrony.service 2024-02-13T03:25:22.747 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T03:25:22.747 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active chrony.service 2024-02-13T03:25:22.753 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:25:22.753 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-13T03:25:22.759 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: enabled 2024-02-13T03:25:22.760 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active chronyd.service 2024-02-13T03:25:22.766 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: active 2024-02-13T03:25:22.767 INFO:teuthology.orchestra.run.smithi080.stderr:Unit chronyd.service is enabled and running 2024-02-13T03:25:22.767 INFO:teuthology.orchestra.run.smithi080.stderr:Repeating the final host check... 2024-02-13T03:25:22.767 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-13T03:25:22.787 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 4.8.3 2024-02-13T03:25:22.802 INFO:teuthology.orchestra.run.smithi080.stderr:podman|docker (/bin/podman) is present 2024-02-13T03:25:22.802 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl is present 2024-02-13T03:25:22.802 INFO:teuthology.orchestra.run.smithi080.stderr:lvcreate is present 2024-02-13T03:25:22.802 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled chrony.service 2024-02-13T03:25:22.808 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T03:25:22.808 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active chrony.service 2024-02-13T03:25:22.814 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:25:22.814 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-13T03:25:22.820 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: enabled 2024-02-13T03:25:22.820 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active chronyd.service 2024-02-13T03:25:22.826 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: active 2024-02-13T03:25:22.827 INFO:teuthology.orchestra.run.smithi080.stderr:Unit chronyd.service is enabled and running 2024-02-13T03:25:22.827 INFO:teuthology.orchestra.run.smithi080.stderr:Host looks OK 2024-02-13T03:25:22.827 INFO:teuthology.orchestra.run.smithi080.stderr:Cluster fsid: 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:22.827 INFO:teuthology.orchestra.run.smithi080.stderr:Acquiring lock 139809763322344 on /run/cephadm/84391b2a-ca1f-11ee-95b9-87774f69a715.lock 2024-02-13T03:25:22.827 INFO:teuthology.orchestra.run.smithi080.stderr:Lock 139809763322344 acquired on /run/cephadm/84391b2a-ca1f-11ee-95b9-87774f69a715.lock 2024-02-13T03:25:22.828 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying IP 172.21.15.80 port 3300 ... 2024-02-13T03:25:22.828 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying IP 172.21.15.80 port 6789 ... 2024-02-13T03:25:22.828 INFO:teuthology.orchestra.run.smithi080.stderr:Base mon IP is 172.21.15.80, final addrv is [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-02-13T03:25:22.828 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /sbin/ip route ls 2024-02-13T03:25:22.830 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.80 metric 100 2024-02-13T03:25:22.831 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.80 metric 100 2024-02-13T03:25:22.831 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /sbin/ip -6 route ls 2024-02-13T03:25:22.834 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-13T03:25:22.834 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-13T03:25:22.834 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-02-13T03:25:22.834 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /sbin/ip -6 addr ls 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cba9/64 scope link 2024-02-13T03:25:22.836 INFO:teuthology.orchestra.run.smithi080.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-13T03:25:22.837 INFO:teuthology.orchestra.run.smithi080.stderr:Mon IP 172.21.15.80 is in CIDR network 172.21.0.0/20 2024-02-13T03:25:22.837 INFO:teuthology.orchestra.run.smithi080.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-13T03:25:22.838 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-13T03:25:22.838 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-13T03:25:22.935 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-13T03:25:29.479 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Getting image source signatures 2024-02-13T03:25:29.494 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-13T03:25:29.495 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-13T03:25:41.347 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-13T03:25:41.506 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: Writing manifest to image destination 2024-02-13T03:25:41.800 INFO:teuthology.orchestra.run.smithi080.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-13T03:25:41.815 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-13T03:25:42.464 INFO:teuthology.orchestra.run.smithi080.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-13T03:25:43.119 INFO:teuthology.orchestra.run.smithi080.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-13T03:25:43.119 INFO:teuthology.orchestra.run.smithi080.stderr:Extracting ceph user uid/gid from container image... 2024-02-13T03:25:43.119 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-13T03:25:43.568 INFO:teuthology.orchestra.run.smithi080.stderr:stat: 167 167 2024-02-13T03:25:44.089 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial keys... 2024-02-13T03:25:44.089 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-13T03:25:44.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQA44cpltkloHRAAWr2ay79/Y0YNYFdS5rgW9A== 2024-02-13T03:25:45.171 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-13T03:25:45.715 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQA54cplqg6uIxAA+AiHZ5eq0wFpsu7jB9gAoA== 2024-02-13T03:25:46.210 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-13T03:25:46.696 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-authtool: AQA64cplk7rKJBAAbBOJ9s2CveBLWw7Jc/FhlA== 2024-02-13T03:25:47.179 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial monmap... 2024-02-13T03:25:47.179 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmptgs_7tgb:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 --addv smithi080 [v2:172.21.15.80:3300,v1:172.21.15.80:6789] /tmp/monmap 2024-02-13T03:25:47.769 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T03:25:47.770 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:47.770 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr:monmaptool for smithi080 [v2:172.21.15.80:3300,v1:172.21.15.80:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: set fsid to 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr:Creating mon... 2024-02-13T03:25:48.358 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/mon.smithi080:/var/lib/ceph/mon/ceph-smithi080:z -v /tmp/ceph-tmpulspbrwx:/tmp/keyring:z -v /tmp/ceph-tmptgs_7tgb:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi080 --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.822+0000 7faad1f65700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.822+0000 7faad1f65700 1 imported monmap: 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: last_changed 2024-02-13T03:25:47.662167+0000 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: created 2024-02-13T03:25:47.662167+0000 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.smithi080 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.945 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.822+0000 7faad1f65700 0 /usr/bin/ceph-mon: set fsid to 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: RocksDB version: 6.8.1 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Compile date May 13 2021 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: DB SUMMARY 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi080/store.db dir, Total Num: 0, files: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi080/store.db: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.error_if_exists: 0 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.create_if_missing: 1 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.paranoid_checks: 1 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.env: 0x560f0c5371c0 2024-02-13T03:25:48.946 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.fs: Posix File System 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.info_log: 0x560f0ec34ec0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.statistics: (nil) 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.use_fsync: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_log_file_size: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.allow_fallocate: 1 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.use_direct_reads: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-13T03:25:48.947 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.db_log_dir: 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi080/store.db 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_subcompactions: 1 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_background_flushes: -1 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.write_buffer_manager: 0x560f0ec3b530 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T03:25:48.948 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.unordered_write: 0 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.row_cache: None 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.wal_filter: None 2024-02-13T03:25:48.949 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.preserve_deletes: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.two_write_queues: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.atomic_flush: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.log_readahead_size: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_background_jobs: 2 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_background_compactions: -1 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T03:25:48.950 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.max_open_files: -1 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Compression algorithms supported: 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kZSTD supported: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kXpressCompression supported: 0 2024-02-13T03:25:48.951 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kLZ4Compression supported: 1 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kBZip2Compression supported: 0 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kZlibCompression supported: 1 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: kSnappyCompression supported: 1 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.823+0000 7faad1f65700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi080/store.db/MANIFEST-000001 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.merge_operator: 2024-02-13T03:25:48.952 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_filter: None 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f0eb4dcb8) 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-13T03:25:48.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_cache: 0x560f0eb82d10 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-13T03:25:48.954 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression: NoCompression 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T03:25:48.955 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.num_levels: 7 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T03:25:48.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T03:25:48.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T03:25:48.958 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.table_properties_collectors: 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.inplace_update_support: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.bloom_locality: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.max_successive_merges: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-13T03:25:48.959 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.ttl: 2592000 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi080/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.824+0000 7faad1f65700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.825+0000 7faad1f65700 4 rocksdb: DB pointer 0x560f0ec47800 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.825+0000 7faabadfe700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.825+0000 7faabadfe700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T03:25:48.960 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:48.961 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:48.962 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:48.963 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.826+0000 7faad1f65700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.827+0000 7faad1f65700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-13T03:25:48.964 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph-mon: debug 2024-02-13T03:25:48.827+0000 7faad1f65700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi080 for mon.smithi080 2024-02-13T03:25:49.513 INFO:teuthology.orchestra.run.smithi080.stderr:create mon.smithi080 on 2024-02-13T03:25:49.513 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:49.519 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl disable ceph.target 2024-02-13T03:25:49.526 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-13T03:25:49.612 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl enable ceph.target 2024-02-13T03:25:49.618 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-13T03:25:49.697 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl start ceph.target 2024-02-13T03:25:49.705 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl enable ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target 2024-02-13T03:25:49.711 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target. 2024-02-13T03:25:49.711 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target. 2024-02-13T03:25:49.790 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target 2024-02-13T03:25:49.797 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl daemon-reload 2024-02-13T03:25:49.879 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:25:49.909 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl reset-failed ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:25:49.915 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to reset failed state of unit ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service: Unit ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service not loaded. 2024-02-13T03:25:49.915 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl enable ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:25:49.922 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target.wants/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service → /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@.service. 2024-02-13T03:25:50.005 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:25:50.538 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-13T03:25:50.546 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-02-13T03:25:50.547 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active firewalld.service 2024-02-13T03:25:50.553 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:25:50.554 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-02-13T03:25:50.554 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T03:25:50.554 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-13T03:25:50.555 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mon to start... 2024-02-13T03:25:50.555 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mon... 2024-02-13T03:25:50.555 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/mon.smithi080:/var/lib/ceph/mon/ceph-smithi080:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-13T03:25:50.836 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:50 smithi080 bash[38983]: c47b4e988c8c11dd05828f6d81cc15225a53e7fe4a6c81fd8ef839e813cf02d2 2024-02-13T03:25:50.837 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:50 smithi080 systemd[1]: Started Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715. 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.144+0000 7f2f7b355700 1 Processor -- start 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.144+0000 7f2f7b355700 1 -- start start 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f7b355700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f7b355700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740fc8e0 con 0x7f2f74070390 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f790f1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f790f1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33432/0 (socket says 172.21.15.80:33432) 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f790f1700 1 -- 172.21.15.80:0/196563905 learned_addr learned my addr 172.21.15.80:0/196563905 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.145+0000 7f2f790f1700 1 -- 172.21.15.80:0/196563905 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f741013e0 con 0x7f2f74070390 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f790f1700 1 --2- 172.21.15.80:0/196563905 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2f6400b5f0 tx=0x7f2f64006ee0).ready entity=mon.0 client_cookie=44d65ecc187b182d server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:51.148 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f63fff700 1 -- 172.21.15.80:0/196563905 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6400d600 con 0x7f2f74070390 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f63fff700 1 -- 172.21.15.80:0/196563905 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400d760 con 0x7f2f74070390 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f63fff700 1 -- 172.21.15.80:0/196563905 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6400da30 con 0x7f2f74070390 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 -- 172.21.15.80:0/196563905 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 msgr2=0x7f2f74100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 --2- 172.21.15.80:0/196563905 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2f6400b5f0 tx=0x7f2f64006ee0).stop 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 -- 172.21.15.80:0/196563905 shutdown_connections 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 --2- 172.21.15.80:0/196563905 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f74100ef0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 -- 172.21.15.80:0/196563905 >> 172.21.15.80:0/196563905 conn(0x7f2f740f6530 msgr2=0x7f2f740f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 -- 172.21.15.80:0/196563905 shutdown_connections 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.146+0000 7f2f7b355700 1 -- 172.21.15.80:0/196563905 wait complete. 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f7b355700 1 Processor -- start 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f7b355700 1 -- start start 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f7b355700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:51.149 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f7b355700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7410ca40 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f790f1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f790f1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33434/0 (socket says 172.21.15.80:33434) 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.147+0000 7f2f790f1700 1 -- 172.21.15.80:0/674579814 learned_addr learned my addr 172.21.15.80:0/674579814 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f790f1700 1 -- 172.21.15.80:0/674579814 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6400b200 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f790f1700 1 --2- 172.21.15.80:0/674579814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2f64006c70 tx=0x7f2f64016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f64011b40 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64011ca0 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7410cc40 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f6401d430 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7410d0d0 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2f6400f070 con 0x7f2f74070390 2024-02-13T03:25:51.151 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.148+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2f64027460 con 0x7f2f74070390 2024-02-13T03:25:51.152 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.149+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f74045ad0 con 0x7f2f74070390 2024-02-13T03:25:51.152 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.151+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2f64016070 con 0x7f2f74070390 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.218+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2f74106190 con 0x7f2f74070390 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.218+0000 7f2f61ffb700 1 -- 172.21.15.80:0/674579814 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f2f64036070 con 0x7f2f74070390 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: cluster: 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: id: 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: services: 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi080 (age 0.741506s) 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-13T03:25:51.221 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: data: 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: pgs: 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 msgr2=0x7f2f7410c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 --2- 172.21.15.80:0/674579814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2f64006c70 tx=0x7f2f64016000).stop 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 shutdown_connections 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 --2- 172.21.15.80:0/674579814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2f74070390 0x7f2f7410c550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 >> 172.21.15.80:0/674579814 conn(0x7f2f740f6530 msgr2=0x7f2f740f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 shutdown_connections 2024-02-13T03:25:51.222 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:51.219+0000 7f2f7b355700 1 -- 172.21.15.80:0/674579814 wait complete. 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: cluster 2024-02-13T03:25:50.478134+0000 mon.smithi080 (mon.0) 2 : cluster [INF] mon.smithi080 is new leader, mons smithi080 in quorum (ranks 0) 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: cluster 2024-02-13T03:25:50.478615+0000 mon.smithi080 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi080=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]} 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: cluster 2024-02-13T03:25:50.481576+0000 mon.smithi080 (mon.0) 4 : cluster [DBG] fsmap 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: cluster 2024-02-13T03:25:50.484169+0000 mon.smithi080 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: cluster 2024-02-13T03:25:50.484550+0000 mon.smithi080 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-13T03:25:51.549 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: audit 2024-02-13T03:25:51.219612+0000 mon.smithi080 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.80:0/674579814' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-13T03:25:51.761 INFO:teuthology.orchestra.run.smithi080.stderr:mon is available 2024-02-13T03:25:51.761 INFO:teuthology.orchestra.run.smithi080.stderr:Assimilating anything we can from ceph.conf... 2024-02-13T03:25:51.761 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z -v /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/mon.smithi080:/var/lib/ceph/mon/ceph-smithi080:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi080/config 2024-02-13T03:25:52.260 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56f6b5700 1 Processor -- start 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56f6b5700 1 -- start start 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56f6b5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56f6b5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680fef80 con 0x7fd5680703f0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56d451700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56d451700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33444/0 (socket says 172.21.15.80:33444) 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56d451700 1 -- 172.21.15.80:0/3587375233 learned_addr learned my addr 172.21.15.80:0/3587375233 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.258+0000 7fd56d451700 1 -- 172.21.15.80:0/3587375233 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680ff0c0 con 0x7fd5680703f0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56d451700 1 --2- 172.21.15.80:0/3587375233 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd56400b5f0 tx=0x7fd564006ee0).ready entity=mon.0 client_cookie=c709ee809dee0f4c server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd557fff700 1 -- 172.21.15.80:0/3587375233 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56400d600 con 0x7fd5680703f0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd557fff700 1 -- 172.21.15.80:0/3587375233 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56400d760 con 0x7fd5680703f0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd557fff700 1 -- 172.21.15.80:0/3587375233 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56400da30 con 0x7fd5680703f0 2024-02-13T03:25:52.261 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 -- 172.21.15.80:0/3587375233 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 msgr2=0x7fd5680fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 --2- 172.21.15.80:0/3587375233 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd56400b5f0 tx=0x7fd564006ee0).stop 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 -- 172.21.15.80:0/3587375233 shutdown_connections 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 --2- 172.21.15.80:0/3587375233 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd5680fea90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 -- 172.21.15.80:0/3587375233 >> 172.21.15.80:0/3587375233 conn(0x7fd5680f6530 msgr2=0x7fd5680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 -- 172.21.15.80:0/3587375233 shutdown_connections 2024-02-13T03:25:52.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.259+0000 7fd56f6b5700 1 -- 172.21.15.80:0/3587375233 wait complete. 2024-02-13T03:25:52.263 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56f6b5700 1 Processor -- start 2024-02-13T03:25:52.263 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56f6b5700 1 -- start start 2024-02-13T03:25:52.263 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56f6b5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:52.263 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56f6b5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56810a820 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56d451700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56d451700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33450/0 (socket says 172.21.15.80:33450) 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56d451700 1 -- 172.21.15.80:0/2549004872 learned_addr learned my addr 172.21.15.80:0/2549004872 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.260+0000 7fd56d451700 1 -- 172.21.15.80:0/2549004872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56400b200 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd56d451700 1 --2- 172.21.15.80:0/2549004872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd564010000 tx=0x7fd564016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd564011c80 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564011de0 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56810aa20 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56401d5b0 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.261+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56810aeb0 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.262+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd56400f070 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.262+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd564027460 con 0x7fd5680703f0 2024-02-13T03:25:52.264 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.262+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd568045ad0 con 0x7fd5680703f0 2024-02-13T03:25:52.265 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.264+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd564016070 con 0x7fd5680703f0 2024-02-13T03:25:52.328 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.327+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd56810c3c0 con 0x7fd5680703f0 2024-02-13T03:25:52.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.330+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd56401d710 con 0x7fd5680703f0 2024-02-13T03:25:52.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.330+0000 7fd555ffb700 1 -- 172.21.15.80:0/2549004872 <== mon.0 v2:172.21.15.80:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7fd564031a70 con 0x7fd5680703f0 2024-02-13T03:25:52.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [global] 2024-02-13T03:25:52.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: fsid = 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.80:3300,v1:172.21.15.80:6789] 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [mgr] 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: [osd] 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-13T03:25:52.335 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 msgr2=0x7fd56810a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 --2- 172.21.15.80:0/2549004872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd564010000 tx=0x7fd564016000).stop 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 shutdown_connections 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 --2- 172.21.15.80:0/2549004872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5680703f0 0x7fd56810a330 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 >> 172.21.15.80:0/2549004872 conn(0x7fd5680f6530 msgr2=0x7fd5680f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 shutdown_connections 2024-02-13T03:25:52.336 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:52.332+0000 7fd56f6b5700 1 -- 172.21.15.80:0/2549004872 wait complete. 2024-02-13T03:25:52.664 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: audit 2024-02-13T03:25:52.328551+0000 mon.smithi080 (mon.0) 8 : audit [INF] from='client.? 172.21.15.80:0/2549004872' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T03:25:52.664 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: audit 2024-02-13T03:25:52.331631+0000 mon.smithi080 (mon.0) 9 : audit [INF] from='client.? 172.21.15.80:0/2549004872' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T03:25:52.942 INFO:teuthology.orchestra.run.smithi080.stderr:Generating new minimal ceph.conf... 2024-02-13T03:25:52.942 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z -v /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/mon.smithi080:/var/lib/ceph/mon/ceph-smithi080:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi080/config 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.458+0000 7ff07c96c700 1 Processor -- start 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.458+0000 7ff07c96c700 1 -- start start 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.458+0000 7ff07c96c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.458+0000 7ff07c96c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078070790 con 0x7ff078070370 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33462/0 (socket says 172.21.15.80:33462) 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07659c700 1 -- 172.21.15.80:0/1463513761 learned_addr learned my addr 172.21.15.80:0/1463513761 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:53.463 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07659c700 1 -- 172.21.15.80:0/1463513761 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0780ff000 con 0x7ff078070370 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07659c700 1 --2- 172.21.15.80:0/1463513761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff06c00b5f0 tx=0x7ff06c006ee0).ready entity=mon.0 client_cookie=64888c466f424f3f server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07559a700 1 -- 172.21.15.80:0/1463513761 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c00d600 con 0x7ff078070370 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07559a700 1 -- 172.21.15.80:0/1463513761 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff06c00dbf0 con 0x7ff078070370 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.459+0000 7ff07559a700 1 -- 172.21.15.80:0/1463513761 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c011cb0 con 0x7ff078070370 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 -- 172.21.15.80:0/1463513761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 msgr2=0x7ff0780feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 --2- 172.21.15.80:0/1463513761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff06c00b5f0 tx=0x7ff06c006ee0).stop 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 -- 172.21.15.80:0/1463513761 shutdown_connections 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 --2- 172.21.15.80:0/1463513761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780feb10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 -- 172.21.15.80:0/1463513761 >> 172.21.15.80:0/1463513761 conn(0x7ff0780f6530 msgr2=0x7ff0780f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 -- 172.21.15.80:0/1463513761 shutdown_connections 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.460+0000 7ff07c96c700 1 -- 172.21.15.80:0/1463513761 wait complete. 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07c96c700 1 Processor -- start 2024-02-13T03:25:53.464 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07c96c700 1 -- start start 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07c96c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07c96c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0780faa10 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33478/0 (socket says 172.21.15.80:33478) 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07659c700 1 -- 172.21.15.80:0/3496173390 learned_addr learned my addr 172.21.15.80:0/3496173390 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07659c700 1 -- 172.21.15.80:0/3496173390 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c00b200 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07659c700 1 --2- 172.21.15.80:0/3496173390 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff06c006800 tx=0x7ff06c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c01b570 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff06c01bb60 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.461+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0780fabb0 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.462+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c01a9f0 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.462+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0780fb040 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.462+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff06c00f070 con 0x7ff078070370 2024-02-13T03:25:53.465 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.462+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff06c025810 con 0x7ff078070370 2024-02-13T03:25:53.466 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.463+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff078045ad0 con 0x7ff078070370 2024-02-13T03:25:53.466 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.465+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff06c016070 con 0x7ff078070370 2024-02-13T03:25:53.529 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.528+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff0780008d0 con 0x7ff078070370 2024-02-13T03:25:53.529 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.529+0000 7ff066ffd700 1 -- 172.21.15.80:0/3496173390 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff06c01ab50 con 0x7ff078070370 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.529+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 msgr2=0x7ff0780fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.529+0000 7ff07c96c700 1 --2- 172.21.15.80:0/3496173390 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff06c006800 tx=0x7ff06c006ee0).stop 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.530+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 shutdown_connections 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.530+0000 7ff07c96c700 1 --2- 172.21.15.80:0/3496173390 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff078070370 0x7ff0780fc590 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.530+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 >> 172.21.15.80:0/3496173390 conn(0x7ff0780f6530 msgr2=0x7ff0780f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.530+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 shutdown_connections 2024-02-13T03:25:53.531 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:53.530+0000 7ff07c96c700 1 -- 172.21.15.80:0/3496173390 wait complete. 2024-02-13T03:25:53.880 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:53 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: audit 2024-02-13T03:25:53.530183+0000 mon.smithi080 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.80:0/3496173390' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T03:25:54.142 INFO:teuthology.orchestra.run.smithi080.stderr:Restarting the monitor... 2024-02-13T03:25:54.142 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl restart ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:25:54.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 systemd[1]: Stopping Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715... 2024-02-13T03:25:54.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: debug 2024-02-13T03:25:54.298+0000 7f1fb6dc5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi080 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T03:25:54.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[38997]: debug 2024-02-13T03:25:54.298+0000 7f1fb6dc5700 -1 mon.smithi080@0(leader) e1 *** Got Signal Terminated *** 2024-02-13T03:25:54.948 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 podman[39321]: ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080 2024-02-13T03:25:55.252 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 systemd[1]: ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service: Succeeded. 2024-02-13T03:25:55.252 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 systemd[1]: Stopped Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715. 2024-02-13T03:25:55.252 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:54 smithi080 systemd[1]: Starting Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715... 2024-02-13T03:25:55.407 INFO:teuthology.orchestra.run.smithi080.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-13T03:25:55.408 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.348+0000 7ff23f6d6700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.348+0000 7ff23f6d6700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.348+0000 7ff23f6d6700 0 pidfile_write: ignore empty --pid-file 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.352+0000 7ff23f6d6700 0 load: jerasure load: lrc load: isa 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: RocksDB version: 6.8.1 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Compile date May 13 2021 2024-02-13T03:25:55.629 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: DB SUMMARY 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: CURRENT file: CURRENT 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi080/store.db dir, Total Num: 1, files: 000004.sst 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi080/store.db: 000006.log size: 98704 ; 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.error_if_exists: 0 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.create_if_missing: 0 2024-02-13T03:25:55.630 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.paranoid_checks: 1 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.env: 0x560b9a69f1c0 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.fs: Posix File System 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.info_log: 0x560b9b483960 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.statistics: (nil) 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.use_fsync: 0 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_log_file_size: 0 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.allow_fallocate: 1 2024-02-13T03:25:55.631 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.use_direct_reads: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.db_log_dir: 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi080/store.db 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_subcompactions: 1 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_background_flushes: -1 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T03:25:55.632 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.write_buffer_manager: 0x560b9c0f8030 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-13T03:25:55.633 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.unordered_write: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.row_cache: None 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.wal_filter: None 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.preserve_deletes: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.two_write_queues: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.atomic_flush: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T03:25:55.634 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.log_readahead_size: 0 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_background_jobs: 2 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_background_compactions: -1 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_open_files: -1 2024-02-13T03:25:55.635 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Compression algorithms supported: 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kZSTD supported: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kXpressCompression supported: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kLZ4Compression supported: 1 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kBZip2Compression supported: 0 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kZlibCompression supported: 1 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: kSnappyCompression supported: 1 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T03:25:55.636 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi080/store.db/MANIFEST-000005 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.merge_operator: 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_filter: None 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b9b39bcb0) 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cache_index_and_filter_blocks: 1 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: pin_top_level_index_and_filter: 1 2024-02-13T03:25:55.637 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: index_type: 0 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: data_block_index_type: 0 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: index_shortening: 1 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: data_block_hash_table_util_ratio: 0.750000 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: hash_index_allow_collision: 1 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: checksum: 1 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: no_block_cache: 0 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_cache: 0x560b9b3d0f10 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_cache_name: BinnedLRUCache 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_cache_options: 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: capacity : 536870912 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: num_shard_bits : 4 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: strict_capacity_limit : 0 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: high_pri_pool_ratio: 0.000 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_cache_compressed: (nil) 2024-02-13T03:25:55.638 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: persistent_cache: (nil) 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_size: 4096 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_size_deviation: 10 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_restart_interval: 16 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: index_block_restart_interval: 1 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: metadata_block_size: 4096 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: partition_filters: 0 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: use_delta_encoding: 1 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: whole_key_filtering: 1 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: verify_compression: 0 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: read_amp_bytes_per_bit: 0 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: format_version: 2 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: enable_index_compression: 1 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: block_align: 0 2024-02-13T03:25:55.639 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression: NoCompression 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.num_levels: 7 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T03:25:55.640 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T03:25:55.641 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-13T03:25:55.642 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T03:25:55.643 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T03:25:55.644 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T03:25:55.644 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T03:25:55.644 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T03:25:55.644 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T03:25:55.644 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.table_properties_collectors: 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.inplace_update_support: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.bloom_locality: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.max_successive_merges: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.ttl: 2592000 2024-02-13T03:25:55.645 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.353+0000 7ff23f6d6700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi080/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707794755355220, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707794755356037, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94278, "table_properties": {"data_size": 92366, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85397, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707794755, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.354+0000 7ff23f6d6700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.355+0000 7ff23f6d6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707794755356428, "job": 1, "event": "recovery_finished"} 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.355+0000 7ff23f6d6700 4 rocksdb: DB pointer 0x560b9b495800 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.356+0000 7ff2264ee700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-13T03:25:55.646 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.356+0000 7ff2264ee700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** DB Stats ** 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** Compaction Stats [default] ** 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.647 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** Compaction Stats [default] ** 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T03:25:55.648 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Cumulative compaction: 0.00 GB write, 34.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Interval compaction: 0.00 GB write, 34.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** Compaction Stats [default] ** 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** Compaction Stats [default] ** 2024-02-13T03:25:55.649 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.1 0.00 0.00 1 0.001 0 0 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: AddFile(Keys): cumulative 0, interval 0 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Cumulative compaction: 0.00 GB write, 34.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.650 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ** File Read Latency Histogram By Level [default] ** 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.356+0000 7ff23f6d6700 0 starting mon.smithi080 rank 0 at public addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] at bind addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi080 fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 1 mon.smithi080@-1(???) e1 preinit fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).mds e1 new map 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).mds e1 print_map 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: e1 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: legacy client fscid: -1 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: No filesystems configured 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T03:25:55.651 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 0 mon.smithi080@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:25:53.530183+0000 mon.smithi080 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.80:0/3496173390' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:25:55.357+0000 7ff23f6d6700 1 mon.smithi080@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 bash[39394]: 004379f27f44168486df0adf06df0c14861861d719748b4e45592565cefa5191 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 systemd[1]: Started Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715. 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:25:55.374629+0000 mon.smithi080 (mon.0) 1 : cluster [INF] mon.smithi080 is new leader, mons smithi080 in quorum (ranks 0) 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:25:55.374689+0000 mon.smithi080 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi080=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0]} 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:25:55.374717+0000 mon.smithi080 (mon.0) 3 : cluster [DBG] fsmap 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:25:55.374739+0000 mon.smithi080 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-13T03:25:55.652 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:25:55.375087+0000 mon.smithi080 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-13T03:25:55.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52c098a700 1 Processor -- start 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52c098a700 1 -- start start 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52c098a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52c098a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0fede0 con 0x7f52bc0703d0 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.882+0000 7f52bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33484/0 (socket says 172.21.15.80:33484) 2024-02-13T03:25:55.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52bb7fe700 1 -- 172.21.15.80:0/3902534519 learned_addr learned my addr 172.21.15.80:0/3902534519 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52bb7fe700 1 -- 172.21.15.80:0/3902534519 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0fef20 con 0x7f52bc0703d0 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52bb7fe700 1 --2- 172.21.15.80:0/3902534519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f52ac00b5f0 tx=0x7f52ac006ee0).ready entity=mon.0 client_cookie=f90b1deead8bc5ae server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52ba7fc700 1 -- 172.21.15.80:0/3902534519 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ac00d600 con 0x7f52bc0703d0 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52ba7fc700 1 -- 172.21.15.80:0/3902534519 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f52ac00dbf0 con 0x7f52bc0703d0 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.883+0000 7f52ba7fc700 1 -- 172.21.15.80:0/3902534519 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ac011cb0 con 0x7f52bc0703d0 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- 172.21.15.80:0/3902534519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 msgr2=0x7f52bc0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 --2- 172.21.15.80:0/3902534519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f52ac00b5f0 tx=0x7f52ac006ee0).stop 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- 172.21.15.80:0/3902534519 shutdown_connections 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 --2- 172.21.15.80:0/3902534519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc0fe8f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- 172.21.15.80:0/3902534519 >> 172.21.15.80:0/3902534519 conn(0x7f52bc0f6390 msgr2=0x7f52bc0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- 172.21.15.80:0/3902534519 shutdown_connections 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- 172.21.15.80:0/3902534519 wait complete. 2024-02-13T03:25:55.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 Processor -- start 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- start start 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.884+0000 7f52c098a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc10ae90 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33486/0 (socket says 172.21.15.80:33486) 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52bb7fe700 1 -- 172.21.15.80:0/2876378380 learned_addr learned my addr 172.21.15.80:0/2876378380 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52bb7fe700 1 -- 172.21.15.80:0/2876378380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac00b200 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52bb7fe700 1 --2- 172.21.15.80:0/2876378380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f52ac006a20 tx=0x7f52ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ac01b570 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f52ac01bb60 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc10b090 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ac01aa20 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc10b520 con 0x7f52bc0703d0 2024-02-13T03:25:55.888 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.885+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f52ac00f070 con 0x7f52bc0703d0 2024-02-13T03:25:55.889 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.886+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f52ac025de0 con 0x7f52bc0703d0 2024-02-13T03:25:55.889 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.886+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc045ad0 con 0x7f52bc0703d0 2024-02-13T03:25:55.889 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.888+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f52ac016070 con 0x7f52bc0703d0 2024-02-13T03:25:55.953 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.953+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f52bc0ffc60 con 0x7f52bc0703d0 2024-02-13T03:25:55.956 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.955+0000 7f52a3fff700 1 -- 172.21.15.80:0/2876378380 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f52ac025460 con 0x7f52bc0703d0 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 msgr2=0x7f52bc10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 --2- 172.21.15.80:0/2876378380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f52ac006a20 tx=0x7f52ac006ee0).stop 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 shutdown_connections 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 --2- 172.21.15.80:0/2876378380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52bc0703d0 0x7f52bc10a9a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 >> 172.21.15.80:0/2876378380 conn(0x7f52bc0f6390 msgr2=0x7f52bc0f7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 shutdown_connections 2024-02-13T03:25:55.957 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:55.956+0000 7f52c098a700 1 -- 172.21.15.80:0/2876378380 wait complete. 2024-02-13T03:25:56.572 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-13T03:25:56.572 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-13T03:25:56.572 INFO:teuthology.orchestra.run.smithi080.stderr:Creating mgr... 2024-02-13T03:25:56.572 INFO:teuthology.orchestra.run.smithi080.stderr:Verifying port 9283 ... 2024-02-13T03:25:56.574 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl daemon-reload 2024-02-13T03:25:56.667 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg 2024-02-13T03:25:56.694 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl reset-failed ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg 2024-02-13T03:25:56.700 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Failed to reset failed state of unit ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg.service: Unit ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg.service not loaded. 2024-02-13T03:25:56.700 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl enable ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg 2024-02-13T03:25:56.706 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: Created symlink /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715.target.wants/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg.service → /etc/systemd/system/ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@.service. 2024-02-13T03:25:56.787 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mgr.smithi080.vyzlfg 2024-02-13T03:25:57.390 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:25:55.956428+0000 mon.smithi080 (mon.0) 6 : audit [INF] from='client.? 172.21.15.80:0/2876378380' entity='client.admin' 2024-02-13T03:25:57.394 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-13T03:25:57.402 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-02-13T03:25:57.403 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active firewalld.service 2024-02-13T03:25:57.410 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:25:57.411 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-02-13T03:25:57.411 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to enable service . firewalld.service is not available 2024-02-13T03:25:57.411 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-13T03:25:57.411 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-13T03:25:57.418 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-02-13T03:25:57.419 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active firewalld.service 2024-02-13T03:25:57.426 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:25:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-02-13T03:25:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-13T03:25:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr to start... 2024-02-13T03:25:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr... 2024-02-13T03:25:57.427 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.990+0000 7f0291cfb700 1 Processor -- start 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.990+0000 7f0291cfb700 1 -- start start 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f0291cfb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f0291cfb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0fef80 con 0x7f028c0703f0 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f028b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f028b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33512/0 (socket says 172.21.15.80:33512) 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f028b7fe700 1 -- 172.21.15.80:0/3199182332 learned_addr learned my addr 172.21.15.80:0/3199182332 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:57.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.991+0000 7f028b7fe700 1 -- 172.21.15.80:0/3199182332 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028c0ff0c0 con 0x7f028c0703f0 2024-02-13T03:25:57.994 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.992+0000 7f028b7fe700 1 --2- 172.21.15.80:0/3199182332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f027c00b5f0 tx=0x7f027c006ee0).ready entity=mon.0 client_cookie=91b78b130b3b4d81 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:57.994 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.992+0000 7f028a7fc700 1 -- 172.21.15.80:0/3199182332 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c00d600 con 0x7f028c0703f0 2024-02-13T03:25:57.994 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.992+0000 7f028a7fc700 1 -- 172.21.15.80:0/3199182332 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f027c00dbf0 con 0x7f028c0703f0 2024-02-13T03:25:57.994 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.992+0000 7f028a7fc700 1 -- 172.21.15.80:0/3199182332 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c011ce0 con 0x7f028c0703f0 2024-02-13T03:25:57.995 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 -- 172.21.15.80:0/3199182332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 msgr2=0x7f028c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:57.995 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 --2- 172.21.15.80:0/3199182332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f027c00b5f0 tx=0x7f027c006ee0).stop 2024-02-13T03:25:57.995 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 -- 172.21.15.80:0/3199182332 shutdown_connections 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 --2- 172.21.15.80:0/3199182332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fea90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 -- 172.21.15.80:0/3199182332 >> 172.21.15.80:0/3199182332 conn(0x7f028c0f6530 msgr2=0x7f028c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 -- 172.21.15.80:0/3199182332 shutdown_connections 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 -- 172.21.15.80:0/3199182332 wait complete. 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.993+0000 7f0291cfb700 1 Processor -- start 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f0291cfb700 1 -- start start 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f0291cfb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f0291cfb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028c0fa990 con 0x7f028c0703f0 2024-02-13T03:25:57.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f028b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f028b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:33514/0 (socket says 172.21.15.80:33514) 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f028b7fe700 1 -- 172.21.15.80:0/2476141521 learned_addr learned my addr 172.21.15.80:0/2476141521 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f028b7fe700 1 -- 172.21.15.80:0/2476141521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c00b200 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f028b7fe700 1 --2- 172.21.15.80:0/2476141521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f027c011ed0 tx=0x7f027c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.994+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c01b570 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f027c01bb60 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028c0fab30 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f027c01aad0 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c0fafc0 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f027c00f070 con 0x7f028c0703f0 2024-02-13T03:25:57.997 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.995+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f027c025d00 con 0x7f028c0703f0 2024-02-13T03:25:57.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:57.996+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028c045ad0 con 0x7f028c0703f0 2024-02-13T03:25:58.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.000+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f027c016070 con 0x7f028c0703f0 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.094+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f028c104b00 con 0x7f028c0703f0 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.094+0000 7f0273fff700 1 -- 172.21.15.80:0/2476141521 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f027c025460 con 0x7f028c0703f0 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "84391b2a-ca1f-11ee-95b9-87774f69a715", 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T03:25:58.102 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "smithi080" 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.103 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T03:25:58.104 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": false, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T03:25:58.105 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-02-13T03:25:50.479944+0000", 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.096+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 msgr2=0x7f028c0fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.096+0000 7f0291cfb700 1 --2- 172.21.15.80:0/2476141521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f027c011ed0 tx=0x7f027c006ee0).stop 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.096+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 shutdown_connections 2024-02-13T03:25:58.106 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.096+0000 7f0291cfb700 1 --2- 172.21.15.80:0/2476141521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f028c0703f0 0x7f028c0fc510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:25:58.107 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.096+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 >> 172.21.15.80:0/2476141521 conn(0x7f028c0f6530 msgr2=0x7f028c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:25:58.107 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.097+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 shutdown_connections 2024-02-13T03:25:58.107 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:25:58.097+0000 7f0291cfb700 1 -- 172.21.15.80:0/2476141521 wait complete. 2024-02-13T03:25:58.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:25:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:25:58.095890+0000 mon.smithi080 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.80:0/2476141521' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T03:25:59.275 INFO:teuthology.orchestra.run.smithi080.stderr:mgr not available, waiting (1/15)... 2024-02-13T03:26:01.277 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-13T03:26:01.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.836+0000 7f6c09777700 1 Processor -- start 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.836+0000 7f6c09777700 1 -- start start 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c09777700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c09777700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040fc7b0 con 0x7f6c040fbea0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c02ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c02ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:35892/0 (socket says 172.21.15.80:35892) 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c02ffd700 1 -- 172.21.15.80:0/2712630583 learned_addr learned my addr 172.21.15.80:0/2712630583 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.837+0000 7f6c02ffd700 1 -- 172.21.15.80:0/2712630583 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040fc8f0 con 0x7f6c040fbea0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c02ffd700 1 --2- 172.21.15.80:0/2712630583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6bf400b5f0 tx=0x7f6bf4006ee0).ready entity=mon.0 client_cookie=95676d1f1cbeccc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c01ffb700 1 -- 172.21.15.80:0/2712630583 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf400d600 con 0x7f6c040fbea0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c01ffb700 1 -- 172.21.15.80:0/2712630583 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6bf400dbf0 con 0x7f6c040fbea0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c01ffb700 1 -- 172.21.15.80:0/2712630583 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf4011cb0 con 0x7f6c040fbea0 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 -- 172.21.15.80:0/2712630583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 msgr2=0x7f6c040fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:01.839 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 --2- 172.21.15.80:0/2712630583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6bf400b5f0 tx=0x7f6bf4006ee0).stop 2024-02-13T03:26:01.840 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 -- 172.21.15.80:0/2712630583 shutdown_connections 2024-02-13T03:26:01.840 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 --2- 172.21.15.80:0/2712630583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040fc2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:01.840 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 -- 172.21.15.80:0/2712630583 >> 172.21.15.80:0/2712630583 conn(0x7f6c040f76a0 msgr2=0x7f6c040f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:01.840 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 -- 172.21.15.80:0/2712630583 shutdown_connections 2024-02-13T03:26:01.840 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.838+0000 7f6c09777700 1 -- 172.21.15.80:0/2712630583 wait complete. 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.839+0000 7f6c09777700 1 Processor -- start 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.839+0000 7f6c09777700 1 -- start start 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.839+0000 7f6c09777700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.839+0000 7f6c09777700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c04070a60 con 0x7f6c040fbea0 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.839+0000 7f6c02ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c02ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:35906/0 (socket says 172.21.15.80:35906) 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c02ffd700 1 -- 172.21.15.80:0/852550704 learned_addr learned my addr 172.21.15.80:0/852550704 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c02ffd700 1 -- 172.21.15.80:0/852550704 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf400b200 con 0x7f6c040fbea0 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c02ffd700 1 --2- 172.21.15.80:0/852550704 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bf400bf40 tx=0x7f6bf4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:01.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf401b570 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c04070c00 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c04071090 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6bf401bb60 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf401aa20 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6bf400f070 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.840+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6bf4025e30 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.841+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be8004fa0 con 0x7f6c040fbea0 2024-02-13T03:26:01.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.844+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6bf4016070 con 0x7f6c040fbea0 2024-02-13T03:26:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.912+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6be8005670 con 0x7f6c040fbea0 2024-02-13T03:26:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.912+0000 7f6bfb7fe700 1 -- 172.21.15.80:0/852550704 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6bf401bcd0 con 0x7f6c040fbea0 2024-02-13T03:26:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:01.918 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "84391b2a-ca1f-11ee-95b9-87774f69a715", 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "smithi080" 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T03:26:01.919 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T03:26:01.920 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.921 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": false, 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-02-13T03:25:50.479944+0000", 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:26:01.922 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 msgr2=0x7f6c040725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 --2- 172.21.15.80:0/852550704 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bf400bf40 tx=0x7f6bf4006ee0).stop 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 shutdown_connections 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 --2- 172.21.15.80:0/852550704 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6c040fbea0 0x7f6c040725e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 >> 172.21.15.80:0/852550704 conn(0x7f6c040f76a0 msgr2=0x7f6c040f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 shutdown_connections 2024-02-13T03:26:01.923 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:01.913+0000 7f6c09777700 1 -- 172.21.15.80:0/852550704 wait complete. 2024-02-13T03:26:02.286 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:01 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:01.912792+0000 mon.smithi080 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.80:0/852550704' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T03:26:02.523 INFO:teuthology.orchestra.run.smithi080.stderr:mgr not available, waiting (2/15)... 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:02.245555+0000 mon.smithi080 (mon.0) 9 : cluster [INF] Activating manager daemon smithi080.vyzlfg 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:02.250940+0000 mon.smithi080 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi080.vyzlfg(active, starting, since 0.00546552s) 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.251675+0000 mon.smithi080 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.251801+0000 mon.smithi080 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.251905+0000 mon.smithi080 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.252010+0000 mon.smithi080 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata", "id": "smithi080"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.252223+0000 mon.smithi080 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mgr metadata", "who": "smithi080.vyzlfg", "id": "smithi080.vyzlfg"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:02.265412+0000 mon.smithi080 (mon.0) 16 : cluster [INF] Manager daemon smithi080.vyzlfg is now available 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.273817+0000 mon.smithi080 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.276951+0000 mon.smithi080 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.278125+0000 mon.smithi080 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:26:03.441 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.282461+0000 mon.smithi080 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:03.442 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:02.284157+0000 mon.smithi080 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.80:0/3878826535' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:04.525 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-13T03:26:04.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:03.253657+0000 mon.smithi080 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi080.vyzlfg(active, since 1.00818s) 2024-02-13T03:26:06.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:05.254401+0000 mon.smithi080 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi080.vyzlfg(active, since 3s) 2024-02-13T03:26:08.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:07.898537+0000 mon.smithi080 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.80:0/3131692303' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T03:26:08.532 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.685+0000 7f966354e700 1 Processor -- start 2024-02-13T03:26:08.532 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.685+0000 7f966354e700 1 -- start start 2024-02-13T03:26:08.532 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f966354e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:08.532 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f966354e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0fc8a0 con 0x7f965c0703d0 2024-02-13T03:26:08.532 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f96612ea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f96612ea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:35952/0 (socket says 172.21.15.80:35952) 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f96612ea700 1 -- 172.21.15.80:0/2767824521 learned_addr learned my addr 172.21.15.80:0/2767824521 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f96612ea700 1 -- 172.21.15.80:0/2767824521 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c1013a0 con 0x7f965c0703d0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f96612ea700 1 --2- 172.21.15.80:0/2767824521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f964c00b5f0 tx=0x7f964c006ee0).ready entity=mon.0 client_cookie=d3288d6f4a1c59d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f964bfff700 1 -- 172.21.15.80:0/2767824521 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c00d600 con 0x7f965c0703d0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f964bfff700 1 -- 172.21.15.80:0/2767824521 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f964c00dbf0 con 0x7f965c0703d0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.686+0000 7f964bfff700 1 -- 172.21.15.80:0/2767824521 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c011cb0 con 0x7f965c0703d0 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- 172.21.15.80:0/2767824521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 msgr2=0x7f965c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 --2- 172.21.15.80:0/2767824521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f964c00b5f0 tx=0x7f964c006ee0).stop 2024-02-13T03:26:08.533 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- 172.21.15.80:0/2767824521 shutdown_connections 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 --2- 172.21.15.80:0/2767824521 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c100eb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- 172.21.15.80:0/2767824521 >> 172.21.15.80:0/2767824521 conn(0x7f965c0f6530 msgr2=0x7f965c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- 172.21.15.80:0/2767824521 shutdown_connections 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- 172.21.15.80:0/2767824521 wait complete. 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 Processor -- start 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.687+0000 7f966354e700 1 -- start start 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f966354e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f966354e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c10d2a0 con 0x7f965c0703d0 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f96612ea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f96612ea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:35968/0 (socket says 172.21.15.80:35968) 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f96612ea700 1 -- 172.21.15.80:0/3131692303 learned_addr learned my addr 172.21.15.80:0/3131692303 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f96612ea700 1 -- 172.21.15.80:0/3131692303 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c00b200 con 0x7f965c0703d0 2024-02-13T03:26:08.534 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f96612ea700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f964c006a20 tx=0x7f964c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c01b570 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f964c01bb60 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c10d4a0 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c01aa20 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.688+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c10e0e0 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.689+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f964c00f070 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.689+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c10d930 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.689+0000 7f9649ffb700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 0x7f96440322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.689+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f964c041170 con 0x7f965c0703d0 2024-02-13T03:26:08.535 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.689+0000 7f9660ae9700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 0x7f96440322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.690+0000 7f9660ae9700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 0x7f96440322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f96580096a0 tx=0x7f9658006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.694+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f964c0228b0 con 0x7f965c0703d0 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.897+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f965c10ef10 con 0x7f965c0703d0 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.897+0000 7f9649ffb700 1 -- 172.21.15.80:0/3131692303 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f964c016030 con 0x7f965c0703d0 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 msgr2=0x7f96440322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 0x7f96440322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f96580096a0 tx=0x7f9658006b40).stop 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 msgr2=0x7f965c10cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f964c006a20 tx=0x7f964c006ee0).stop 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 shutdown_connections 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f964402fe10 0x7f96440322d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:08.536 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 --2- 172.21.15.80:0/3131692303 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f965c0703d0 0x7f965c10cdb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 >> 172.21.15.80:0/3131692303 conn(0x7f965c0f6530 msgr2=0x7f965c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 shutdown_connections 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:07.899+0000 7f966354e700 1 -- 172.21.15.80:0/3131692303 wait complete. 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsid": "84391b2a-ca1f-11ee-95b9-87774f69a715", 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "health": { 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "checks": {}, 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mutes": [] 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum": [ 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 0 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "smithi080" 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "monmap": { 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:08.538 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osdmap": { 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.539 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgmap": { 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "fsmap": { 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-13T03:26:08.540 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modules": [ 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "iostat", 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "restful" 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ], 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "servicemap": { 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "modified": "2024-02-13T03:25:50.479944+0000", 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "services": {} 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: }, 2024-02-13T03:26:08.541 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-13T03:26:08.542 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:13.119 INFO:teuthology.orchestra.run.smithi080.stderr:mgr is available 2024-02-13T03:26:13.119 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling cephadm module... 2024-02-13T03:26:13.119 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-13T03:26:15.544 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b1d163700 1 Processor -- start 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b1d163700 1 -- start start 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b1d163700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b1d163700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180fb440 con 0x7f8b180fab30 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:56298/0 (socket says 172.21.15.80:56298) 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b17fff700 1 -- 172.21.15.80:0/3136606562 learned_addr learned my addr 172.21.15.80:0/3136606562 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.541+0000 7f8b17fff700 1 -- 172.21.15.80:0/3136606562 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180fb580 con 0x7f8b180fab30 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b17fff700 1 --2- 172.21.15.80:0/3136606562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8b0800b5f0 tx=0x7f8b08006ee0).ready entity=mon.0 client_cookie=4351e4755d953478 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b16ffd700 1 -- 172.21.15.80:0/3136606562 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b0800d600 con 0x7f8b180fab30 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b16ffd700 1 -- 172.21.15.80:0/3136606562 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8b0800dbf0 con 0x7f8b180fab30 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b16ffd700 1 -- 172.21.15.80:0/3136606562 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b08011cb0 con 0x7f8b180fab30 2024-02-13T03:26:15.545 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 -- 172.21.15.80:0/3136606562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 msgr2=0x7f8b180faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 --2- 172.21.15.80:0/3136606562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8b0800b5f0 tx=0x7f8b08006ee0).stop 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 -- 172.21.15.80:0/3136606562 shutdown_connections 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 --2- 172.21.15.80:0/3136606562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b180faf50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 -- 172.21.15.80:0/3136606562 >> 172.21.15.80:0/3136606562 conn(0x7f8b180f6390 msgr2=0x7f8b180f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 -- 172.21.15.80:0/3136606562 shutdown_connections 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.542+0000 7f8b1d163700 1 -- 172.21.15.80:0/3136606562 wait complete. 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b1d163700 1 Processor -- start 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b1d163700 1 -- start start 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b1d163700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b1d163700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1810aea0 con 0x7f8b180fab30 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.543+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:56308/0 (socket says 172.21.15.80:56308) 2024-02-13T03:26:15.546 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b17fff700 1 -- 172.21.15.80:0/693663517 learned_addr learned my addr 172.21.15.80:0/693663517 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:15.547 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b17fff700 1 -- 172.21.15.80:0/693663517 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0800b200 con 0x7f8b180fab30 2024-02-13T03:26:15.547 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b17fff700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8b08002f40 tx=0x7f8b08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:15.547 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b0801b410 con 0x7f8b180fab30 2024-02-13T03:26:15.547 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8b0801ba00 con 0x7f8b180fab30 2024-02-13T03:26:15.547 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b0801a900 con 0x7f8b180fab30 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1810b0a0 con 0x7f8b180fab30 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.544+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1810b530 con 0x7f8b180fab30 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f8b0800f070 con 0x7f8b180fab30 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b14ff9700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 0x7f8b040322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8b08041200 con 0x7f8b180fab30 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b177fe700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 0x7f8b040322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b177fe700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 0x7f8b040322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8b0c0096a0 tx=0x7f8b0c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:15.548 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.545+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b18045ad0 con 0x7f8b180fab30 2024-02-13T03:26:15.550 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.550+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b08016070 con 0x7f8b180fab30 2024-02-13T03:26:15.735 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:15.734+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8b18104a80 con 0x7f8b180fab30 2024-02-13T03:26:16.367 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.366+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8b080257e0 con 0x7f8b180fab30 2024-02-13T03:26:16.368 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.367+0000 7f8b14ff9700 1 -- 172.21.15.80:0/693663517 <== mon.0 v2:172.21.15.80:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8b08005d50 con 0x7f8b180fab30 2024-02-13T03:26:16.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 msgr2=0x7f8b040322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:16.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 0x7f8b040322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8b0c0096a0 tx=0x7f8b0c006b40).stop 2024-02-13T03:26:16.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 msgr2=0x7f8b1810a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:16.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8b08002f40 tx=0x7f8b08006ee0).stop 2024-02-13T03:26:16.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 shutdown_connections 2024-02-13T03:26:16.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f8b0402fe10 0x7f8b040322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:16.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 --2- 172.21.15.80:0/693663517 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b180fab30 0x7f8b1810a9b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:16.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 >> 172.21.15.80:0/693663517 conn(0x7f8b180f6390 msgr2=0x7f8b18070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:16.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 shutdown_connections 2024-02-13T03:26:16.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:16.369+0000 7f8b1d163700 1 -- 172.21.15.80:0/693663517 wait complete. 2024-02-13T03:26:16.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:15.736026+0000 mon.smithi080 (mon.0) 25 : audit [INF] from='client.? 172.21.15.80:0/693663517' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-13T03:26:16.997 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe575d1a700 1 Processor -- start 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe575d1a700 1 -- start start 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe575d1a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe575d1a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700fb6a0 con 0x7fe5700fad90 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe56f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe56f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:56324/0 (socket says 172.21.15.80:56324) 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe56f7fe700 1 -- 172.21.15.80:0/1793483928 learned_addr learned my addr 172.21.15.80:0/1793483928 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe56f7fe700 1 -- 172.21.15.80:0/1793483928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700fb7e0 con 0x7fe5700fad90 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.492+0000 7fe56f7fe700 1 --2- 172.21.15.80:0/1793483928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe56000b5f0 tx=0x7fe560006ee0).ready entity=mon.0 client_cookie=89d1ebc0c254000e server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:17.498 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe56e7fc700 1 -- 172.21.15.80:0/1793483928 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56000d600 con 0x7fe5700fad90 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe56e7fc700 1 -- 172.21.15.80:0/1793483928 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe56000dbf0 con 0x7fe5700fad90 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe56e7fc700 1 -- 172.21.15.80:0/1793483928 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe560011cb0 con 0x7fe5700fad90 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 -- 172.21.15.80:0/1793483928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 msgr2=0x7fe5700fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 --2- 172.21.15.80:0/1793483928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe56000b5f0 tx=0x7fe560006ee0).stop 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 -- 172.21.15.80:0/1793483928 shutdown_connections 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 --2- 172.21.15.80:0/1793483928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe5700fb1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 -- 172.21.15.80:0/1793483928 >> 172.21.15.80:0/1793483928 conn(0x7fe5700f6530 msgr2=0x7fe5700f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.493+0000 7fe575d1a700 1 -- 172.21.15.80:0/1793483928 shutdown_connections 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe575d1a700 1 -- 172.21.15.80:0/1793483928 wait complete. 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe575d1a700 1 Processor -- start 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe575d1a700 1 -- start start 2024-02-13T03:26:17.499 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe575d1a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe575d1a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700704b0 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.494+0000 7fe56f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe56f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:56326/0 (socket says 172.21.15.80:56326) 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe56f7fe700 1 -- 172.21.15.80:0/440114943 learned_addr learned my addr 172.21.15.80:0/440114943 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe56f7fe700 1 -- 172.21.15.80:0/440114943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56000b200 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe56f7fe700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe560011ea0 tx=0x7fe560006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56001b570 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe56001bb60 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5700706b0 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56001a8d0 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.495+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57010df30 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.496+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe570045ad0 con 0x7fe5700fad90 2024-02-13T03:26:17.500 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.496+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe56000f070 con 0x7fe5700fad90 2024-02-13T03:26:17.501 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.497+0000 7fe557fff700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 0x7fe55c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:17.501 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.497+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe560041e10 con 0x7fe5700fad90 2024-02-13T03:26:17.501 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.497+0000 7fe56effd700 1 -- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 msgr2=0x7fe55c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:17.501 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.497+0000 7fe56effd700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 0x7fe55c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T03:26:17.501 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.500+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe560022900 con 0x7fe5700fad90 2024-02-13T03:26:17.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:16.368612+0000 mon.smithi080 (mon.0) 26 : audit [INF] from='client.? 172.21.15.80:0/693663517' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-13T03:26:17.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:16.368657+0000 mon.smithi080 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi080.vyzlfg(active, since 14s) 2024-02-13T03:26:17.699 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.698+0000 7fe56effd700 1 -- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 msgr2=0x7fe55c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:17.699 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.698+0000 7fe56effd700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 0x7fe55c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T03:26:17.729 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.727+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe570027a00 con 0x7fe5700fad90 2024-02-13T03:26:17.729 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.728+0000 7fe557fff700 1 -- 172.21.15.80:0/440114943 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe560016070 con 0x7fe5700fad90 2024-02-13T03:26:17.729 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:17.729 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-13T03:26:17.729 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-02-13T03:26:17.730 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "active_name": "smithi080.vyzlfg", 2024-02-13T03:26:17.730 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T03:26:17.730 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 msgr2=0x7fe55c032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 0x7fe55c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 msgr2=0x7fe57010d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe560011ea0 tx=0x7fe560006ee0).stop 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 shutdown_connections 2024-02-13T03:26:17.731 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7fe55c02fe70 0x7fe55c032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:17.732 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 --2- 172.21.15.80:0/440114943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe5700fad90 0x7fe57010d800 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:17.732 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.729+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 >> 172.21.15.80:0/440114943 conn(0x7fe5700f6530 msgr2=0x7fe5700f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:17.732 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.730+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 shutdown_connections 2024-02-13T03:26:17.732 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:17.730+0000 7fe575d1a700 1 -- 172.21.15.80:0/440114943 wait complete. 2024-02-13T03:26:18.280 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for the mgr to restart... 2024-02-13T03:26:18.280 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr epoch 5... 2024-02-13T03:26:18.280 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-13T03:26:18.561 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:17.729051+0000 mon.smithi080 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.80:0/440114943' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T03:26:18.882 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.879+0000 7f36f1843700 1 Processor -- start 2024-02-13T03:26:18.882 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.879+0000 7f36f1843700 1 -- start start 2024-02-13T03:26:18.882 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.879+0000 7f36f1843700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:18.882 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.879+0000 7f36f1843700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec0fef80 con 0x7f36ec0703f0 2024-02-13T03:26:18.882 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50344/0 (socket says 172.21.15.80:50344) 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36eaffd700 1 -- 172.21.15.80:0/938130920 learned_addr learned my addr 172.21.15.80:0/938130920 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36eaffd700 1 -- 172.21.15.80:0/938130920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0ff0c0 con 0x7f36ec0703f0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36eaffd700 1 --2- 172.21.15.80:0/938130920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f36e000b5f0 tx=0x7f36e0006ee0).ready entity=mon.0 client_cookie=5ee31f09268a3d2d server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36e9ffb700 1 -- 172.21.15.80:0/938130920 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36e000d600 con 0x7f36ec0703f0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36e9ffb700 1 -- 172.21.15.80:0/938130920 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f36e000dbf0 con 0x7f36ec0703f0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.880+0000 7f36e9ffb700 1 -- 172.21.15.80:0/938130920 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36e0011ce0 con 0x7f36ec0703f0 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 -- 172.21.15.80:0/938130920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 msgr2=0x7f36ec0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 --2- 172.21.15.80:0/938130920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f36e000b5f0 tx=0x7f36e0006ee0).stop 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 -- 172.21.15.80:0/938130920 shutdown_connections 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 --2- 172.21.15.80:0/938130920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec0fea90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 -- 172.21.15.80:0/938130920 >> 172.21.15.80:0/938130920 conn(0x7f36ec0f6530 msgr2=0x7f36ec0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 -- 172.21.15.80:0/938130920 shutdown_connections 2024-02-13T03:26:18.883 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.881+0000 7f36f1843700 1 -- 172.21.15.80:0/938130920 wait complete. 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36f1843700 1 Processor -- start 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36f1843700 1 -- start start 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36f1843700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36f1843700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec10b040 con 0x7f36ec0703f0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50352/0 (socket says 172.21.15.80:50352) 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.882+0000 7f36eaffd700 1 -- 172.21.15.80:0/3832037674 learned_addr learned my addr 172.21.15.80:0/3832037674 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36eaffd700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e000b200 con 0x7f36ec0703f0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36eaffd700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f36e0011ed0 tx=0x7f36e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36e001b570 con 0x7f36ec0703f0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f36e001bb60 con 0x7f36ec0703f0 2024-02-13T03:26:18.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36e001aad0 con 0x7f36ec0703f0 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec10b240 con 0x7f36ec0703f0 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.883+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec10b6d0 con 0x7f36ec0703f0 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.884+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f36e000f070 con 0x7f36ec0703f0 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.884+0000 7f36d37fe700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.884+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f36e00421c0 con 0x7f36ec0703f0 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.884+0000 7f36ea7fc700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 msgr2=0x7f36d80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.884+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T03:26:18.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:18.885+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36ec059100 con 0x7f36d802fe10 2024-02-13T03:26:19.086 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:19.085+0000 7f36ea7fc700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 msgr2=0x7f36d80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:19.086 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:19.085+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T03:26:19.487 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:19.486+0000 7f36ea7fc700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 msgr2=0x7f36d80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:19.487 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:19.486+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T03:26:20.288 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:20.287+0000 7f36ea7fc700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 msgr2=0x7f36d80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4141954803 2024-02-13T03:26:20.288 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:20.287+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T03:26:21.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:21.370+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f36e0024ca0 con 0x7f36ec0703f0 2024-02-13T03:26:21.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:21.370+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 msgr2=0x7f36d80322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T03:26:21.371 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:21.370+0000 7f36d37fe700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:21.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:21 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:21.278871+0000 mon.smithi080 (mon.0) 29 : cluster [INF] Active manager daemon smithi080.vyzlfg restarted 2024-02-13T03:26:21.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:21 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:21.279237+0000 mon.smithi080 (mon.0) 30 : cluster [INF] Activating manager daemon smithi080.vyzlfg 2024-02-13T03:26:21.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:21 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:21.283243+0000 mon.smithi080 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-13T03:26:22.378 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.377+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f36e0023d60 con 0x7f36ec0703f0 2024-02-13T03:26:22.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.377+0000 7f36d37fe700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 0x7f36d8035b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:22.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.377+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36ec059100 con 0x7f36d8033780 2024-02-13T03:26:22.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.378+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 0x7f36d8035b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:22.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.378+0000 7f36ea7fc700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 0x7f36d8035b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f36dc0036d0 tx=0x7f36dc006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:22.380 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.379+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f36ec059100 con 0x7f36d8033780 2024-02-13T03:26:22.386 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.385+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f36ec045ad0 con 0x7f36d8033780 2024-02-13T03:26:22.386 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36d37fe700 1 -- 172.21.15.80:0/3832037674 <== mgr.14114 v2:172.21.15.80:6800/1174143345 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f36ec045ad0 con 0x7f36d8033780 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 msgr2=0x7f36d8035b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 0x7f36d8035b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f36dc0036d0 tx=0x7f36dc006ca0).stop 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 msgr2=0x7f36ec10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:22.388 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f36e0011ed0 tx=0x7f36e0006ee0).stop 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 shutdown_connections 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f36d8033780 0x7f36d8035b70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36ec0703f0 0x7f36ec10ab50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 --2- 172.21.15.80:0/3832037674 >> [v2:172.21.15.80:6800/4141954803,v1:172.21.15.80:6801/4141954803] conn(0x7f36d802fe10 0x7f36d80322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 >> 172.21.15.80:0/3832037674 conn(0x7f36ec0f6530 msgr2=0x7f36ec0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 shutdown_connections 2024-02-13T03:26:22.389 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:22.386+0000 7f36f1843700 1 -- 172.21.15.80:0/3832037674 wait complete. 2024-02-13T03:26:22.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:21.372027+0000 mon.smithi080 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi080.vyzlfg(active, starting, since 0.0928615s) 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.372875+0000 mon.smithi080 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata", "id": "smithi080"}]: dispatch 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.373462+0000 mon.smithi080 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mgr metadata", "who": "smithi080.vyzlfg", "id": "smithi080.vyzlfg"}]: dispatch 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.374288+0000 mon.smithi080 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.374505+0000 mon.smithi080 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.374708+0000 mon.smithi080 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:21.388558+0000 mon.smithi080 (mon.0) 38 : cluster [INF] Manager daemon smithi080.vyzlfg is now available 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.401643+0000 mon.smithi080 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.404512+0000 mon.smithi080 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.407096+0000 mon.smithi080 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.493670+0000 mon.smithi080 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.494339+0000 mon.smithi080 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:22.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.495746+0000 mon.smithi080 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:22.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.497418+0000 mon.smithi080 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:22.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.507021+0000 mon.smithi080 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:26:22.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:21.508166+0000 mon.smithi080 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:26:22.996 INFO:teuthology.orchestra.run.smithi080.stderr:mgr epoch 5 is available 2024-02-13T03:26:22.996 INFO:teuthology.orchestra.run.smithi080.stderr:Setting orchestrator backend to cephadm... 2024-02-13T03:26:22.996 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.566+0000 7fb88fdb5700 1 Processor -- start 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.566+0000 7fb88fdb5700 1 -- start start 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88fdb5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88fdb5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880fef40 con 0x7fb888070430 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88db51700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88db51700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50410/0 (socket says 172.21.15.80:50410) 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88db51700 1 -- 172.21.15.80:0/1414912478 learned_addr learned my addr 172.21.15.80:0/1414912478 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88db51700 1 -- 172.21.15.80:0/1414912478 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880ff080 con 0x7fb888070430 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88db51700 1 --2- 172.21.15.80:0/1414912478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb87800b5f0 tx=0x7fb878006ee0).ready entity=mon.0 client_cookie=e550b142140bc06f server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88cb4f700 1 -- 172.21.15.80:0/1414912478 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87800d600 con 0x7fb888070430 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88cb4f700 1 -- 172.21.15.80:0/1414912478 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb87800dbf0 con 0x7fb888070430 2024-02-13T03:26:23.571 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.567+0000 7fb88cb4f700 1 -- 172.21.15.80:0/1414912478 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb878011cb0 con 0x7fb888070430 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1414912478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 msgr2=0x7fb8880fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1414912478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb87800b5f0 tx=0x7fb878006ee0).stop 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1414912478 shutdown_connections 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1414912478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb8880fea50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1414912478 >> 172.21.15.80:0/1414912478 conn(0x7fb8880f6530 msgr2=0x7fb8880f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1414912478 shutdown_connections 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.568+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1414912478 wait complete. 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88fdb5700 1 Processor -- start 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88fdb5700 1 -- start start 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88fdb5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88fdb5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88810b080 con 0x7fb888070430 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88db51700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:23.572 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88db51700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50412/0 (socket says 172.21.15.80:50412) 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88db51700 1 -- 172.21.15.80:0/1905812786 learned_addr learned my addr 172.21.15.80:0/1905812786 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88db51700 1 -- 172.21.15.80:0/1905812786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87800b200 con 0x7fb888070430 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.569+0000 7fb88db51700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb87800df90 tx=0x7fb878006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87801b410 con 0x7fb888070430 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb87801ba00 con 0x7fb888070430 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88810b280 con 0x7fb888070430 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87801aa30 con 0x7fb888070430 2024-02-13T03:26:23.573 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88810b710 con 0x7fb888070430 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.570+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888045ad0 con 0x7fb888070430 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.571+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb87800f070 con 0x7fb888070430 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.571+0000 7fb8767fc700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 0x7fb8700322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.571+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb8780414d0 con 0x7fb888070430 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.571+0000 7fb88d350700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 0x7fb8700322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:23.574 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.571+0000 7fb88d350700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 0x7fb8700322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb8840096a0 tx=0x7fb884006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:23.575 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.575+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb878022310 con 0x7fb888070430 2024-02-13T03:26:23.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:23 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:22.380408+0000 mgr.smithi080.vyzlfg (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T03:26:23.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:23 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:22.380866+0000 mon.smithi080 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi080.vyzlfg(active, since 1.1017s) 2024-02-13T03:26:23.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:23 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:22.387001+0000 mgr.smithi080.vyzlfg (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T03:26:23.756 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.755+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb88810c030 con 0x7fb87002fe10 2024-02-13T03:26:23.761 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.760+0000 7fb8767fc700 1 -- 172.21.15.80:0/1905812786 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb88810c030 con 0x7fb87002fe10 2024-02-13T03:26:23.763 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 msgr2=0x7fb8700322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:23.763 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 0x7fb8700322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb8840096a0 tx=0x7fb884006b40).stop 2024-02-13T03:26:23.763 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 msgr2=0x7fb88810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:23.763 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb87800df90 tx=0x7fb878006ee0).stop 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 shutdown_connections 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb87002fe10 0x7fb8700322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 --2- 172.21.15.80:0/1905812786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb888070430 0x7fb88810ab90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.762+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 >> 172.21.15.80:0/1905812786 conn(0x7fb8880f6530 msgr2=0x7fb8880f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.763+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 shutdown_connections 2024-02-13T03:26:23.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:23.763+0000 7fb88fdb5700 1 -- 172.21.15.80:0/1905812786 wait complete. 2024-02-13T03:26:24.343 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-13T03:26:24.598 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:23.380648+0000 mon.smithi080 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi080.vyzlfg(active, since 2s) 2024-02-13T03:26:24.598 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:23.761160+0000 mon.smithi080 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:24.599 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:23.765244+0000 mon.smithi080 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.890+0000 7fb68ceea700 1 Processor -- start 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.890+0000 7fb68ceea700 1 -- start start 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.890+0000 7fb68ceea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.890+0000 7fb68ceea700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6880fc9a0 con 0x7fb688070390 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:24.894 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50420/0 (socket says 172.21.15.80:50420) 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68659c700 1 -- 172.21.15.80:0/1959787285 learned_addr learned my addr 172.21.15.80:0/1959787285 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68659c700 1 -- 172.21.15.80:0/1959787285 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6881014a0 con 0x7fb688070390 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68659c700 1 --2- 172.21.15.80:0/1959787285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb67c00b5f0 tx=0x7fb67c006ee0).ready entity=mon.0 client_cookie=ba89e246faa88a0e server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68559a700 1 -- 172.21.15.80:0/1959787285 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c00d600 con 0x7fb688070390 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68559a700 1 -- 172.21.15.80:0/1959787285 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb67c00dbf0 con 0x7fb688070390 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.891+0000 7fb68559a700 1 -- 172.21.15.80:0/1959787285 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c011cb0 con 0x7fb688070390 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 -- 172.21.15.80:0/1959787285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 msgr2=0x7fb688100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 --2- 172.21.15.80:0/1959787285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb67c00b5f0 tx=0x7fb67c006ee0).stop 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 -- 172.21.15.80:0/1959787285 shutdown_connections 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 --2- 172.21.15.80:0/1959787285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb688100fb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 -- 172.21.15.80:0/1959787285 >> 172.21.15.80:0/1959787285 conn(0x7fb6880f65b0 msgr2=0x7fb6880f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 -- 172.21.15.80:0/1959787285 shutdown_connections 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.892+0000 7fb68ceea700 1 -- 172.21.15.80:0/1959787285 wait complete. 2024-02-13T03:26:24.895 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68ceea700 1 Processor -- start 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68ceea700 1 -- start start 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68ceea700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68ceea700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68810d2c0 con 0x7fb688070390 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50432/0 (socket says 172.21.15.80:50432) 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68659c700 1 -- 172.21.15.80:0/725162065 learned_addr learned my addr 172.21.15.80:0/725162065 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68659c700 1 -- 172.21.15.80:0/725162065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c00b200 con 0x7fb688070390 2024-02-13T03:26:24.896 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.893+0000 7fb68659c700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb67c006800 tx=0x7fb67c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:24.897 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c01b570 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb67c01bb60 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68810d4c0 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c01a9f0 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68810e100 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb67c00f070 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 0x7fb66c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.894+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb67c041480 con 0x7fb688070390 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.895+0000 7fb685d9b700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 0x7fb66c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.895+0000 7fb685d9b700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 0x7fb66c032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb6780096a0 tx=0x7fb678006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:24.898 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.895+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68810d950 con 0x7fb688070390 2024-02-13T03:26:24.900 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:24.899+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb67c01b6d0 con 0x7fb688070390 2024-02-13T03:26:25.050 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.049+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb688102160 con 0x7fb66c02fe70 2024-02-13T03:26:25.051 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.050+0000 7fb676ffd700 1 -- 172.21.15.80:0/725162065 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb688102160 con 0x7fb66c02fe70 2024-02-13T03:26:25.051 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: value unchanged 2024-02-13T03:26:25.053 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 msgr2=0x7fb66c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:25.053 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 0x7fb66c032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb6780096a0 tx=0x7fb678006b40).stop 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 msgr2=0x7fb68810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb67c006800 tx=0x7fb67c006ee0).stop 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 shutdown_connections 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb66c02fe70 0x7fb66c032330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 --2- 172.21.15.80:0/725162065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb688070390 0x7fb68810cdd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 >> 172.21.15.80:0/725162065 conn(0x7fb6880f65b0 msgr2=0x7fb6880f8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 shutdown_connections 2024-02-13T03:26:25.054 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:25.052+0000 7fb68ceea700 1 -- 172.21.15.80:0/725162065 wait complete. 2024-02-13T03:26:25.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:25 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:23.757246+0000 mgr.smithi080.vyzlfg (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:25.725 INFO:teuthology.orchestra.run.smithi080.stderr:Generating ssh key... 2024-02-13T03:26:25.725 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbed02a0700 1 Processor -- start 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbed02a0700 1 -- start start 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbed02a0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbed02a0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec80fef80 con 0x7fbec80703f0 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbece03c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:26.212 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbece03c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50438/0 (socket says 172.21.15.80:50438) 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.210+0000 7fbece03c700 1 -- 172.21.15.80:0/1435967502 learned_addr learned my addr 172.21.15.80:0/1435967502 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbece03c700 1 -- 172.21.15.80:0/1435967502 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec80ff0c0 con 0x7fbec80703f0 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbece03c700 1 --2- 172.21.15.80:0/1435967502 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbeb800b5f0 tx=0x7fbeb8006ee0).ready entity=mon.0 client_cookie=b5f08fd0b1244d3b server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbecd03a700 1 -- 172.21.15.80:0/1435967502 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb800d600 con 0x7fbec80703f0 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbecd03a700 1 -- 172.21.15.80:0/1435967502 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbeb800dbf0 con 0x7fbec80703f0 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbecd03a700 1 -- 172.21.15.80:0/1435967502 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb8011ce0 con 0x7fbec80703f0 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbed02a0700 1 -- 172.21.15.80:0/1435967502 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 msgr2=0x7fbec80fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.211+0000 7fbed02a0700 1 --2- 172.21.15.80:0/1435967502 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbeb800b5f0 tx=0x7fbeb8006ee0).stop 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 -- 172.21.15.80:0/1435967502 shutdown_connections 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 --2- 172.21.15.80:0/1435967502 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fea90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 -- 172.21.15.80:0/1435967502 >> 172.21.15.80:0/1435967502 conn(0x7fbec80f6530 msgr2=0x7fbec80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 -- 172.21.15.80:0/1435967502 shutdown_connections 2024-02-13T03:26:26.214 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 -- 172.21.15.80:0/1435967502 wait complete. 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 Processor -- start 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.212+0000 7fbed02a0700 1 -- start start 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbed02a0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbed02a0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec80fa990 con 0x7fbec80703f0 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbece03c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbece03c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50450/0 (socket says 172.21.15.80:50450) 2024-02-13T03:26:26.215 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbece03c700 1 -- 172.21.15.80:0/927353266 learned_addr learned my addr 172.21.15.80:0/927353266 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbece03c700 1 -- 172.21.15.80:0/927353266 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb800b200 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbece03c700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fbeb8011ed0 tx=0x7fbeb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb801b570 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec80fab30 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbeb801bb60 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb801aad0 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.213+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec80fafc0 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.214+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbeb800f070 con 0x7fbec80703f0 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.214+0000 7fbeb6ffd700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 0x7fbeb00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:26.216 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.214+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbeb8041420 con 0x7fbec80703f0 2024-02-13T03:26:26.217 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.214+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec8045ad0 con 0x7fbec80703f0 2024-02-13T03:26:26.217 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.214+0000 7fbecd83b700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 0x7fbeb00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:26.217 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.215+0000 7fbecd83b700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 0x7fbeb00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbec40096a0 tx=0x7fbec4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:26.219 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.219+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbeb8016070 con 0x7fbec80703f0 2024-02-13T03:26:26.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.368+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbec80fff80 con 0x7fbeb002fe10 2024-02-13T03:26:26.429 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.428+0000 7fbeb6ffd700 1 -- 172.21.15.80:0/927353266 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbec80fff80 con 0x7fbeb002fe10 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 msgr2=0x7fbeb00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 0x7fbeb00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbec40096a0 tx=0x7fbec4006b40).stop 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 msgr2=0x7fbec80fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fbeb8011ed0 tx=0x7fbeb8006ee0).stop 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 shutdown_connections 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fbeb002fe10 0x7fbeb00322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 --2- 172.21.15.80:0/927353266 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbec80703f0 0x7fbec80fc510 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:26.431 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 >> 172.21.15.80:0/927353266 conn(0x7fbec80f6530 msgr2=0x7fbec80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:26.432 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 shutdown_connections 2024-02-13T03:26:26.432 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:26.430+0000 7fbed02a0700 1 -- 172.21.15.80:0/927353266 wait complete. 2024-02-13T03:26:26.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:25.051151+0000 mgr.smithi080.vyzlfg (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:27.083 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-13T03:26:27.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.600+0000 7f032613a700 1 Processor -- start 2024-02-13T03:26:27.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f032613a700 1 -- start start 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f032613a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f032613a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03200fb6a0 con 0x7f03200fad90 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50454/0 (socket says 172.21.15.80:50454) 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f031f7fe700 1 -- 172.21.15.80:0/1609894820 learned_addr learned my addr 172.21.15.80:0/1609894820 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f031f7fe700 1 -- 172.21.15.80:0/1609894820 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200fb7e0 con 0x7f03200fad90 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.601+0000 7f031f7fe700 1 --2- 172.21.15.80:0/1609894820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f031400b5f0 tx=0x7f0314006ee0).ready entity=mon.0 client_cookie=158214cac83e3b65 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f031e7fc700 1 -- 172.21.15.80:0/1609894820 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f031400d600 con 0x7f03200fad90 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f031e7fc700 1 -- 172.21.15.80:0/1609894820 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f031400dbf0 con 0x7f03200fad90 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f031e7fc700 1 -- 172.21.15.80:0/1609894820 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0314011cb0 con 0x7f03200fad90 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f032613a700 1 -- 172.21.15.80:0/1609894820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 msgr2=0x7f03200fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f032613a700 1 --2- 172.21.15.80:0/1609894820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f031400b5f0 tx=0x7f0314006ee0).stop 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f032613a700 1 -- 172.21.15.80:0/1609894820 shutdown_connections 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f032613a700 1 --2- 172.21.15.80:0/1609894820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f03200fb1b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:27.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.602+0000 7f032613a700 1 -- 172.21.15.80:0/1609894820 >> 172.21.15.80:0/1609894820 conn(0x7f03200f6530 msgr2=0x7f03200f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:27.605 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.603+0000 7f032613a700 1 -- 172.21.15.80:0/1609894820 shutdown_connections 2024-02-13T03:26:27.605 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.603+0000 7f032613a700 1 -- 172.21.15.80:0/1609894820 wait complete. 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.603+0000 7f032613a700 1 Processor -- start 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.603+0000 7f032613a700 1 -- start start 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f032613a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f032613a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320070450 con 0x7f03200fad90 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f031f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:50462/0 (socket says 172.21.15.80:50462) 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f031f7fe700 1 -- 172.21.15.80:0/99193935 learned_addr learned my addr 172.21.15.80:0/99193935 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f031f7fe700 1 -- 172.21.15.80:0/99193935 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031400b200 con 0x7f03200fad90 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f031f7fe700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0314011ea0 tx=0x7f0314006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:27.607 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f031401b570 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f031401bb60 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320070650 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f031401a8d0 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.604+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032010df10 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.605+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f031400f070 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.605+0000 7f0307fff700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 0x7f030c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.605+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0314041450 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.605+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0320045ad0 con 0x7f03200fad90 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.605+0000 7f031effd700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 0x7f030c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:27.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.606+0000 7f031effd700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 0x7f030c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f03100096a0 tx=0x7f0310006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:27.610 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.609+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0314016070 con 0x7f03200fad90 2024-02-13T03:26:27.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:26.370419+0000 mgr.smithi080.vyzlfg (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-13T03:26:27.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:26.425927+0000 mon.smithi080 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:27.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:26.428415+0000 mon.smithi080 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:27.762 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.761+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f03200fcc90 con 0x7f030c02fe10 2024-02-13T03:26:27.762 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.761+0000 7f0307fff700 1 -- 172.21.15.80:0/99193935 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0310011400 con 0x7f030c02fe10 2024-02-13T03:26:27.762 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb/Ii27rT46N558ce5+igpaOoTc18zSE9HL3KV+/ElBQpxVIGI9FRiFRFB2zZ+79SLaA0pvoIxJlzo8ZOk/aTDQIL8bgioMS6Ydou9bEg44PZGsTuuREJGdnSUVmg816N2tM4fZxDPUF+Ae/TmXXGL+XawpwX+MwDWEmuX7v0VmoLUnllAVvhB9Qfq90mEZLytYO6og/PlZRnbQD/saqIyV6UAENM1Gw/KqM+/VSi/MxVrNMXBy2C5x0RoxY0JhvJPmiFyDYxwp3h680GXnwZdeK9sgYhdOdCsQgQMJQclwvSAKbi+lT7qdkN5IGyh8+NXPzQ0wLtElVNeyV5vfZv/cMJ5fDLvSw4Kg5M8MzFMVB5wIMWz+qpeTxneZAxEzyO3YP3wrxSf2YIrsII1y005L2CFyjGGR1LXASsdKFW5JDnpaTDzpIiQXoGLK9yfJBTzyou6E6G2yYoIt2u/KaNaL8zxutpnUjmWLXWFrN3IbacDwIBkAAGLZ7x+8+1xfVM= ceph-84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:26:27.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 msgr2=0x7f030c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:27.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 0x7f030c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f03100096a0 tx=0x7f0310006b40).stop 2024-02-13T03:26:27.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 msgr2=0x7f032010d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:27.764 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0314011ea0 tx=0x7f0314006ee0).stop 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 shutdown_connections 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f030c02fe10 0x7f030c0322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 --2- 172.21.15.80:0/99193935 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03200fad90 0x7f032010d860 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 >> 172.21.15.80:0/99193935 conn(0x7f03200f6530 msgr2=0x7f03200f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 shutdown_connections 2024-02-13T03:26:27.765 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:27.763+0000 7f032613a700 1 -- 172.21.15.80:0/99193935 wait complete. 2024-02-13T03:26:28.323 INFO:teuthology.orchestra.run.smithi080.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-13T03:26:28.323 INFO:teuthology.orchestra.run.smithi080.stderr:Adding key to root@localhost authorized_keys... 2024-02-13T03:26:28.324 INFO:teuthology.orchestra.run.smithi080.stderr:Adding host smithi080... 2024-02-13T03:26:28.324 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi080 172.21.15.80 2024-02-13T03:26:28.822 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.819+0000 7f0f9fd4c700 1 Processor -- start 2024-02-13T03:26:28.822 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.819+0000 7f0f9fd4c700 1 -- start start 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9fd4c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9fd4c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980fef80 con 0x7f0f980703f0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9dae8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9dae8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47116/0 (socket says 172.21.15.80:47116) 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9dae8700 1 -- 172.21.15.80:0/36515074 learned_addr learned my addr 172.21.15.80:0/36515074 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9dae8700 1 -- 172.21.15.80:0/36515074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980ff0c0 con 0x7f0f980703f0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9dae8700 1 --2- 172.21.15.80:0/36515074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0f8800b5f0 tx=0x7f0f88006ee0).ready entity=mon.0 client_cookie=a276ff625b17ac2a server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9cae6700 1 -- 172.21.15.80:0/36515074 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f8800d600 con 0x7f0f980703f0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9cae6700 1 -- 172.21.15.80:0/36515074 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f8800dbf0 con 0x7f0f980703f0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.820+0000 7f0f9cae6700 1 -- 172.21.15.80:0/36515074 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f88011ce0 con 0x7f0f980703f0 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/36515074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 msgr2=0x7f0f980fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/36515074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0f8800b5f0 tx=0x7f0f88006ee0).stop 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/36515074 shutdown_connections 2024-02-13T03:26:28.823 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/36515074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f980fea90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:28.824 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/36515074 >> 172.21.15.80:0/36515074 conn(0x7f0f980f6530 msgr2=0x7f0f980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:28.824 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/36515074 shutdown_connections 2024-02-13T03:26:28.824 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.821+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/36515074 wait complete. 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9fd4c700 1 Processor -- start 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9fd4c700 1 -- start start 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9fd4c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9fd4c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9810b010 con 0x7f0f980703f0 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9dae8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9dae8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47128/0 (socket says 172.21.15.80:47128) 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9dae8700 1 -- 172.21.15.80:0/3696063201 learned_addr learned my addr 172.21.15.80:0/3696063201 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9dae8700 1 -- 172.21.15.80:0/3696063201 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8800b200 con 0x7f0f980703f0 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9dae8700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0f88011ed0 tx=0x7f0f88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f8801b570 con 0x7f0f980703f0 2024-02-13T03:26:28.826 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f8801bb60 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f8801aad0 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.822+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9810b210 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.823+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9810b6a0 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.823+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0f8800f070 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.823+0000 7f0f867fc700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 0x7f0f800322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.823+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f98045ad0 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.823+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0f880413c0 con 0x7f0f980703f0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.824+0000 7f0f9d2e7700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 0x7f0f800322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:28.827 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.824+0000 7f0f9d2e7700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 0x7f0f800322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0f940096a0 tx=0x7f0f94006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:28.828 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.828+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f88016070 con 0x7f0f980703f0 2024-02-13T03:26:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:27.762512+0000 mgr.smithi080.vyzlfg (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:28.996 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:28.995+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi080", "addr": "172.21.15.80", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f9810bfc0 con 0x7f0f8002fe10 2024-02-13T03:26:29.580 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Added host 'smithi080' 2024-02-13T03:26:29.581 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.578+0000 7f0f867fc700 1 -- 172.21.15.80:0/3696063201 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f0f9810bfc0 con 0x7f0f8002fe10 2024-02-13T03:26:29.582 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 msgr2=0x7f0f800322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:29.582 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 0x7f0f800322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0f940096a0 tx=0x7f0f94006b40).stop 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 msgr2=0x7f0f9810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0f88011ed0 tx=0x7f0f88006ee0).stop 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 shutdown_connections 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f0f8002fe10 0x7f0f800322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 --2- 172.21.15.80:0/3696063201 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f980703f0 0x7f0f9810ab20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 >> 172.21.15.80:0/3696063201 conn(0x7f0f980f6530 msgr2=0x7f0f980f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 shutdown_connections 2024-02-13T03:26:29.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:29.581+0000 7f0f9fd4c700 1 -- 172.21.15.80:0/3696063201 wait complete. 2024-02-13T03:26:29.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:29 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:28.997496+0000 mgr.smithi080.vyzlfg (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi080", "addr": "172.21.15.80", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:30.230 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying mon service with default placement... 2024-02-13T03:26:30.230 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-13T03:26:30.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:29.578916+0000 mon.smithi080 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:30.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:29.579239+0000 mgr.smithi080.vyzlfg (mgr.14114) 9 : cephadm [INF] Added host smithi080 2024-02-13T03:26:30.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:29.579496+0000 mon.smithi080 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:31.144 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.134+0000 7f86f28ad700 1 Processor -- start 2024-02-13T03:26:31.144 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.134+0000 7f86f28ad700 1 -- start start 2024-02-13T03:26:31.144 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.134+0000 7f86f28ad700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:31.144 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.134+0000 7f86f28ad700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec0fb6a0 con 0x7f86ec0fad90 2024-02-13T03:26:31.144 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86ebfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86ebfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47136/0 (socket says 172.21.15.80:47136) 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86ebfff700 1 -- 172.21.15.80:0/2154829811 learned_addr learned my addr 172.21.15.80:0/2154829811 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86ebfff700 1 -- 172.21.15.80:0/2154829811 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec0fb7e0 con 0x7f86ec0fad90 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86ebfff700 1 --2- 172.21.15.80:0/2154829811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f86e000b5f0 tx=0x7f86e0006ee0).ready entity=mon.0 client_cookie=db7c7f7fa73ecca1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86eaffd700 1 -- 172.21.15.80:0/2154829811 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e000d600 con 0x7f86ec0fad90 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86eaffd700 1 -- 172.21.15.80:0/2154829811 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f86e000dbf0 con 0x7f86ec0fad90 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.135+0000 7f86eaffd700 1 -- 172.21.15.80:0/2154829811 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e0011cb0 con 0x7f86ec0fad90 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 -- 172.21.15.80:0/2154829811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 msgr2=0x7f86ec0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2154829811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f86e000b5f0 tx=0x7f86e0006ee0).stop 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 -- 172.21.15.80:0/2154829811 shutdown_connections 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2154829811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec0fb1b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 -- 172.21.15.80:0/2154829811 >> 172.21.15.80:0/2154829811 conn(0x7f86ec0f6530 msgr2=0x7f86ec0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 -- 172.21.15.80:0/2154829811 shutdown_connections 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 -- 172.21.15.80:0/2154829811 wait complete. 2024-02-13T03:26:31.145 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.136+0000 7f86f28ad700 1 Processor -- start 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86f28ad700 1 -- start start 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86f28ad700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86f28ad700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec10b030 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86ebfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86ebfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47148/0 (socket says 172.21.15.80:47148) 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86ebfff700 1 -- 172.21.15.80:0/2324620172 learned_addr learned my addr 172.21.15.80:0/2324620172 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86ebfff700 1 -- 172.21.15.80:0/2324620172 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e000b200 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86ebfff700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f86e0002f40 tx=0x7f86e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e001b410 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f86e001ba00 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec10b230 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e001a900 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.137+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec10b6c0 con 0x7f86ec0fad90 2024-02-13T03:26:31.146 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.138+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f86e000f070 con 0x7f86ec0fad90 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.138+0000 7f86e8ff9700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 0x7f86d80322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.138+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f86e0041370 con 0x7f86ec0fad90 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.138+0000 7f86eb7fe700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 0x7f86d80322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.138+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec045ad0 con 0x7f86ec0fad90 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.139+0000 7f86eb7fe700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 0x7f86d80322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f86dc0096a0 tx=0x7f86dc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:31.147 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.142+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86e0016070 con 0x7f86ec0fad90 2024-02-13T03:26:31.317 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.316+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f86ec0008d0 con 0x7f86d802fe10 2024-02-13T03:26:31.321 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.320+0000 7f86e8ff9700 1 -- 172.21.15.80:0/2324620172 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f86ec0008d0 con 0x7f86d802fe10 2024-02-13T03:26:31.321 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-13T03:26:31.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 msgr2=0x7f86d80322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:31.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 0x7f86d80322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f86dc0096a0 tx=0x7f86dc006b40).stop 2024-02-13T03:26:31.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 msgr2=0x7f86ec10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:31.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f86e0002f40 tx=0x7f86e0006ee0).stop 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 shutdown_connections 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f86d802fe10 0x7f86d80322d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 --2- 172.21.15.80:0/2324620172 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f86ec0fad90 0x7f86ec10ab40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 >> 172.21.15.80:0/2324620172 conn(0x7f86ec0f6530 msgr2=0x7f86ec0f7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 shutdown_connections 2024-02-13T03:26:31.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:31.322+0000 7f86f28ad700 1 -- 172.21.15.80:0/2324620172 wait complete. 2024-02-13T03:26:31.930 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying mgr service with default placement... 2024-02-13T03:26:31.930 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f9894e7b700 1 Processor -- start 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f9894e7b700 1 -- start start 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f9894e7b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f9894e7b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900fef80 con 0x7f98900703f0 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f988e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:32.409 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f988e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47158/0 (socket says 172.21.15.80:47158) 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f988e59c700 1 -- 172.21.15.80:0/1735206702 learned_addr learned my addr 172.21.15.80:0/1735206702 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.395+0000 7f988e59c700 1 -- 172.21.15.80:0/1735206702 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900ff0c0 con 0x7f98900703f0 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f988e59c700 1 --2- 172.21.15.80:0/1735206702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f988000b5f0 tx=0x7f9880006ee0).ready entity=mon.0 client_cookie=a579d2922bfa4b21 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f988d59a700 1 -- 172.21.15.80:0/1735206702 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988000d600 con 0x7f98900703f0 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f988d59a700 1 -- 172.21.15.80:0/1735206702 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f988000dbf0 con 0x7f98900703f0 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f988d59a700 1 -- 172.21.15.80:0/1735206702 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9880011ce0 con 0x7f98900703f0 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 -- 172.21.15.80:0/1735206702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 msgr2=0x7f98900fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1735206702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f988000b5f0 tx=0x7f9880006ee0).stop 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 -- 172.21.15.80:0/1735206702 shutdown_connections 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1735206702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fea90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 -- 172.21.15.80:0/1735206702 >> 172.21.15.80:0/1735206702 conn(0x7f98900f6530 msgr2=0x7f98900f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 -- 172.21.15.80:0/1735206702 shutdown_connections 2024-02-13T03:26:32.410 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.396+0000 7f9894e7b700 1 -- 172.21.15.80:0/1735206702 wait complete. 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f9894e7b700 1 Processor -- start 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f9894e7b700 1 -- start start 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f9894e7b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f9894e7b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900fa990 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f988e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f988e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47164/0 (socket says 172.21.15.80:47164) 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f988e59c700 1 -- 172.21.15.80:0/1730036833 learned_addr learned my addr 172.21.15.80:0/1730036833 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f988e59c700 1 -- 172.21.15.80:0/1730036833 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988000b200 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.397+0000 7f988e59c700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f98800115c0 tx=0x7f9880006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.398+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988001b410 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.398+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f988001ba00 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.398+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98900fab30 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.398+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f988001aaa0 con 0x7f98900703f0 2024-02-13T03:26:32.411 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.398+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98900fafc0 con 0x7f98900703f0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9890045ad0 con 0x7f98900703f0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f988000f070 con 0x7f98900703f0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f987effd700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 0x7f9874032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f98800417e0 con 0x7f98900703f0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f988dd9b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 0x7f9874032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.399+0000 7f988dd9b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 0x7f9874032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f98840096a0 tx=0x7f9884006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:32.412 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.403+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9880019af0 con 0x7f98900703f0 2024-02-13T03:26:32.576 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.575+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f98900fb200 con 0x7f987402fe70 2024-02-13T03:26:32.580 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.579+0000 7f987effd700 1 -- 172.21.15.80:0/1730036833 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f98900fb200 con 0x7f987402fe70 2024-02-13T03:26:32.580 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 msgr2=0x7f9874032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 0x7f9874032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f98840096a0 tx=0x7f9884006b40).stop 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 msgr2=0x7f98900fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f98800115c0 tx=0x7f9880006ee0).stop 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 shutdown_connections 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f987402fe70 0x7f9874032330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 --2- 172.21.15.80:0/1730036833 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98900703f0 0x7f98900fc510 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 >> 172.21.15.80:0/1730036833 conn(0x7f98900f6530 msgr2=0x7f98900f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 shutdown_connections 2024-02-13T03:26:32.583 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:32.581+0000 7f9894e7b700 1 -- 172.21.15.80:0/1730036833 wait complete. 2024-02-13T03:26:32.607 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:31.318379+0000 mgr.smithi080.vyzlfg (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:32.608 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:31.319129+0000 mgr.smithi080.vyzlfg (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-13T03:26:32.608 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:31.321262+0000 mon.smithi080 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:33.070 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying crash service with default placement... 2024-02-13T03:26:33.070 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-13T03:26:33.602 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d9c6c700 1 Processor -- start 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d9c6c700 1 -- start start 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d9c6c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d9c6c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40fb6a0 con 0x7fc7d40fad90 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47168/0 (socket says 172.21.15.80:47168) 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.599+0000 7fc7d37fe700 1 -- 172.21.15.80:0/837213527 learned_addr learned my addr 172.21.15.80:0/837213527 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d37fe700 1 -- 172.21.15.80:0/837213527 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d40fb7e0 con 0x7fc7d40fad90 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d37fe700 1 --2- 172.21.15.80:0/837213527 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc7c800b5f0 tx=0x7fc7c8006ee0).ready entity=mon.0 client_cookie=ae5aee4f6e3e5184 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d27fc700 1 -- 172.21.15.80:0/837213527 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c800d600 con 0x7fc7d40fad90 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d27fc700 1 -- 172.21.15.80:0/837213527 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc7c800dbf0 con 0x7fc7d40fad90 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d27fc700 1 -- 172.21.15.80:0/837213527 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c8011cb0 con 0x7fc7d40fad90 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/837213527 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 msgr2=0x7fc7d40fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:33.603 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.600+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/837213527 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc7c800b5f0 tx=0x7fc7c8006ee0).stop 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/837213527 shutdown_connections 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/837213527 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d40fb1b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/837213527 >> 172.21.15.80:0/837213527 conn(0x7fc7d40f6530 msgr2=0x7fc7d40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/837213527 shutdown_connections 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/837213527 wait complete. 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 Processor -- start 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 -- start start 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.601+0000 7fc7d9c6c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d9c6c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40704b0 con 0x7fc7d40fad90 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:33.604 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47182/0 (socket says 172.21.15.80:47182) 2024-02-13T03:26:33.605 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d37fe700 1 -- 172.21.15.80:0/2226615296 learned_addr learned my addr 172.21.15.80:0/2226615296 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:33.605 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d37fe700 1 -- 172.21.15.80:0/2226615296 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c800b200 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d37fe700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc7c8011ea0 tx=0x7fc7c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c801b570 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d40706b0 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc7c801bb60 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c801a8d0 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.602+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d410df30 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.603+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc7c800f070 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.603+0000 7fc7bbfff700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 0x7fc7c00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.603+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc7c8041450 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.603+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d4045ad0 con 0x7fc7d40fad90 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.603+0000 7fc7d2ffd700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 0x7fc7c00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:33.606 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.604+0000 7fc7d2ffd700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 0x7fc7c00322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc7c40096a0 tx=0x7fc7c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:33.608 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.607+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7c8016070 con 0x7fc7d40fad90 2024-02-13T03:26:33.784 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.783+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7d4057f10 con 0x7fc7c002fe10 2024-02-13T03:26:33.788 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.787+0000 7fc7bbfff700 1 -- 172.21.15.80:0/2226615296 <== mgr.14114 v2:172.21.15.80:6800/1174143345 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fc7d4057f10 con 0x7fc7c002fe10 2024-02-13T03:26:33.788 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-13T03:26:33.790 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 msgr2=0x7fc7c00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:33.790 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 0x7fc7c00322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc7c40096a0 tx=0x7fc7c4006b40).stop 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 msgr2=0x7fc7d410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc7c8011ea0 tx=0x7fc7c8006ee0).stop 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 shutdown_connections 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fc7c002fe10 0x7fc7c00322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 --2- 172.21.15.80:0/2226615296 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7d40fad90 0x7fc7d410d800 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 >> 172.21.15.80:0/2226615296 conn(0x7fc7d40f6530 msgr2=0x7fc7d40f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 shutdown_connections 2024-02-13T03:26:33.791 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:33.789+0000 7fc7d9c6c700 1 -- 172.21.15.80:0/2226615296 wait complete. 2024-02-13T03:26:33.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:32.577593+0000 mgr.smithi080.vyzlfg (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:33.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:32.578112+0000 mgr.smithi080.vyzlfg (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-13T03:26:33.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:32.580387+0000 mon.smithi080 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:34.444 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling mgr prometheus module... 2024-02-13T03:26:34.444 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-13T03:26:34.999 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.997+0000 7fb8ccf5b700 1 Processor -- start 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.997+0000 7fb8ccf5b700 1 -- start start 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8ccf5b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8ccf5b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80fef80 con 0x7fb8c80703f0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47186/0 (socket says 172.21.15.80:47186) 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c659c700 1 -- 172.21.15.80:0/801821363 learned_addr learned my addr 172.21.15.80:0/801821363 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c659c700 1 -- 172.21.15.80:0/801821363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80ff0c0 con 0x7fb8c80703f0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c659c700 1 --2- 172.21.15.80:0/801821363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8bc00b5f0 tx=0x7fb8bc006ee0).ready entity=mon.0 client_cookie=dc1ffcee7fb09ab9 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c559a700 1 -- 172.21.15.80:0/801821363 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc00d600 con 0x7fb8c80703f0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c559a700 1 -- 172.21.15.80:0/801821363 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb8bc00dbf0 con 0x7fb8c80703f0 2024-02-13T03:26:35.000 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.998+0000 7fb8c559a700 1 -- 172.21.15.80:0/801821363 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc011ce0 con 0x7fb8c80703f0 2024-02-13T03:26:35.001 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/801821363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 msgr2=0x7fb8c80fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:35.001 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/801821363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb8bc00b5f0 tx=0x7fb8bc006ee0).stop 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/801821363 shutdown_connections 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/801821363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c80fea90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/801821363 >> 172.21.15.80:0/801821363 conn(0x7fb8c80f6530 msgr2=0x7fb8c80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/801821363 shutdown_connections 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:34.999+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/801821363 wait complete. 2024-02-13T03:26:35.003 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8ccf5b700 1 Processor -- start 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8ccf5b700 1 -- start start 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8ccf5b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8ccf5b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c810b010 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8c659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8c659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47190/0 (socket says 172.21.15.80:47190) 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8c659c700 1 -- 172.21.15.80:0/1667354564 learned_addr learned my addr 172.21.15.80:0/1667354564 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8c659c700 1 -- 172.21.15.80:0/1667354564 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc00b200 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.000+0000 7fb8c659c700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb8bc011ed0 tx=0x7fb8bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc01b570 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb8bc01bb60 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c810b210 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc01aad0 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c810b6a0 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb8bc00f070 con 0x7fb8c80703f0 2024-02-13T03:26:35.004 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 0x7fb8ac0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:35.005 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb8bc01bcd0 con 0x7fb8c80703f0 2024-02-13T03:26:35.005 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.001+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8045ad0 con 0x7fb8c80703f0 2024-02-13T03:26:35.005 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.002+0000 7fb8c5d9b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 0x7fb8ac0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:35.005 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.002+0000 7fb8c5d9b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 0x7fb8ac0322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:35.007 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.006+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8bc016070 con 0x7fb8c80703f0 2024-02-13T03:26:35.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:33.785496+0000 mgr.smithi080.vyzlfg (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:35.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:33.786127+0000 mgr.smithi080.vyzlfg (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-13T03:26:35.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:33.788279+0000 mon.smithi080 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.80:0/4200970686' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:35.191 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.191+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fb8c80ffe00 con 0x7fb8c80703f0 2024-02-13T03:26:35.792 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.790+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb8bc0682b0 con 0x7fb8c80703f0 2024-02-13T03:26:35.792 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.792+0000 7fb8b6ffd700 1 -- 172.21.15.80:0/1667354564 <== mon.0 v2:172.21.15.80:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fb8bc019d10 con 0x7fb8c80703f0 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 msgr2=0x7fb8ac0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 0x7fb8ac0322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).stop 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 msgr2=0x7fb8c810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb8bc011ed0 tx=0x7fb8bc006ee0).stop 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 shutdown_connections 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7fb8ac02fe10 0x7fb8ac0322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 --2- 172.21.15.80:0/1667354564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8c80703f0 0x7fb8c810ab20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 >> 172.21.15.80:0/1667354564 conn(0x7fb8c80f6530 msgr2=0x7fb8c80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:35.796 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 shutdown_connections 2024-02-13T03:26:35.797 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:35.794+0000 7fb8ccf5b700 1 -- 172.21.15.80:0/1667354564 wait complete. 2024-02-13T03:26:36.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:35 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:35.192576+0000 mon.smithi080 (mon.0) 59 : audit [INF] from='client.? 172.21.15.80:0/1667354564' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T03:26:36.417 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying prometheus service with default placement... 2024-02-13T03:26:36.417 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.929+0000 7f94d03b3700 1 Processor -- start 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.929+0000 7f94d03b3700 1 -- start start 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94d03b3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94d03b3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80fc8e0 con 0x7f94c8070390 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94ce14f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94ce14f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47210/0 (socket says 172.21.15.80:47210) 2024-02-13T03:26:36.931 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94ce14f700 1 -- 172.21.15.80:0/3988773062 learned_addr learned my addr 172.21.15.80:0/3988773062 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:36.934 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94ce14f700 1 -- 172.21.15.80:0/3988773062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c81013e0 con 0x7f94c8070390 2024-02-13T03:26:36.934 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94ce14f700 1 --2- 172.21.15.80:0/3988773062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f94c400b5f0 tx=0x7f94c4006ee0).ready entity=mon.0 client_cookie=2dc25084295f8464 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:36.934 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.930+0000 7f94cd14d700 1 -- 172.21.15.80:0/3988773062 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c400d600 con 0x7f94c8070390 2024-02-13T03:26:36.934 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94cd14d700 1 -- 172.21.15.80:0/3988773062 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f94c400dbf0 con 0x7f94c8070390 2024-02-13T03:26:36.934 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94cd14d700 1 -- 172.21.15.80:0/3988773062 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c4011cb0 con 0x7f94c8070390 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 -- 172.21.15.80:0/3988773062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 msgr2=0x7f94c8100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3988773062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f94c400b5f0 tx=0x7f94c4006ee0).stop 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 -- 172.21.15.80:0/3988773062 shutdown_connections 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3988773062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c8100ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 -- 172.21.15.80:0/3988773062 >> 172.21.15.80:0/3988773062 conn(0x7f94c80f6530 msgr2=0x7f94c80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 -- 172.21.15.80:0/3988773062 shutdown_connections 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.931+0000 7f94d03b3700 1 -- 172.21.15.80:0/3988773062 wait complete. 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94d03b3700 1 Processor -- start 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94d03b3700 1 -- start start 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94d03b3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94d03b3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c810d270 con 0x7f94c8070390 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94ce14f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94ce14f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:47212/0 (socket says 172.21.15.80:47212) 2024-02-13T03:26:36.935 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94ce14f700 1 -- 172.21.15.80:0/3024866621 learned_addr learned my addr 172.21.15.80:0/3024866621 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94ce14f700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c400b200 con 0x7f94c8070390 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.932+0000 7f94ce14f700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f94c4006800 tx=0x7f94c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.933+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c401b570 con 0x7f94c8070390 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.933+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f94c401bb60 con 0x7f94c8070390 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.933+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c810d470 con 0x7f94c8070390 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.933+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c401a9f0 con 0x7f94c8070390 2024-02-13T03:26:36.936 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.933+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c810e080 con 0x7f94c8070390 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f94c400f070 con 0x7f94c8070390 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94b6ffd700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f94c4041430 con 0x7f94c8070390 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94cd94e700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/1174143345 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T03:26:36.937 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.934+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c8059100 con 0x7f94c8070390 2024-02-13T03:26:36.939 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:36.938+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94c4016070 con 0x7f94c8070390 2024-02-13T03:26:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:35.792869+0000 mon.smithi080 (mon.0) 60 : audit [INF] from='client.? 172.21.15.80:0/1667354564' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T03:26:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:35.792951+0000 mon.smithi080 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi080.vyzlfg(active, since 14s) 2024-02-13T03:26:37.136 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:37.135+0000 7f94cd94e700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/1174143345 2024-02-13T03:26:37.136 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:37.135+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T03:26:37.157 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:37.156+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f94c81020d0 con 0x7f94b002fec0 2024-02-13T03:26:37.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:37.536+0000 7f94cd94e700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/1174143345 2024-02-13T03:26:37.537 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:37.536+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T03:26:38.338 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:38.337+0000 7f94cd94e700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/1174143345 2024-02-13T03:26:38.338 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:38.337+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T03:26:39.939 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:39.938+0000 7f94cd94e700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/1174143345 2024-02-13T03:26:39.940 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:39.938+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-13T03:26:41.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:41.368+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f94c401bcd0 con 0x7f94c8070390 2024-02-13T03:26:41.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:41.368+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 msgr2=0x7f94b0032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T03:26:41.370 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:41.368+0000 7f94b6ffd700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:40.741899+0000 mon.smithi080 (mon.0) 62 : cluster [INF] Active manager daemon smithi080.vyzlfg restarted 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:40.742340+0000 mon.smithi080 (mon.0) 63 : cluster [INF] Activating manager daemon smithi080.vyzlfg 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:40.743913+0000 mon.smithi080 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:41.371217+0000 mon.smithi080 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi080.vyzlfg(active, starting, since 0.628974s) 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.372471+0000 mon.smithi080 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata", "id": "smithi080"}]: dispatch 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.373212+0000 mon.smithi080 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mgr metadata", "who": "smithi080.vyzlfg", "id": "smithi080.vyzlfg"}]: dispatch 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.374255+0000 mon.smithi080 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.374438+0000 mon.smithi080 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.374592+0000 mon.smithi080 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:41.388695+0000 mon.smithi080 (mon.0) 71 : cluster [INF] Manager daemon smithi080.vyzlfg is now available 2024-02-13T03:26:41.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.496318+0000 mon.smithi080 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:41.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.497774+0000 mon.smithi080 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:41.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.513493+0000 mon.smithi080 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:26:41.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:41.516892+0000 mon.smithi080 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:26:42.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.370+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mon.0 v2:172.21.15.80:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f94c4043130 con 0x7f94c8070390 2024-02-13T03:26:42.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.370+0000 7f94b6ffd700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 0x7f94b00360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:42.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.370+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 --> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f94c81020d0 con 0x7f94b0033ce0 2024-02-13T03:26:42.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.370+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 0x7f94b00360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:42.372 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.371+0000 7f94cd94e700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 0x7f94b00360d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f94b8006fd0 tx=0x7f94b8006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:42.376 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.375+0000 7f94b6ffd700 1 -- 172.21.15.80:0/3024866621 <== mgr.14142 v2:172.21.15.80:6800/4099640074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f94c81020d0 con 0x7f94b0033ce0 2024-02-13T03:26:42.376 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-13T03:26:42.378 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 msgr2=0x7f94b00360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 0x7f94b00360d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f94b8006fd0 tx=0x7f94b8006d30).stop 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 msgr2=0x7f94c810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f94c4006800 tx=0x7f94c4006ee0).stop 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 shutdown_connections 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f94b0033ce0 0x7f94b00360d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94c8070390 0x7f94c810cd80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 --2- 172.21.15.80:0/3024866621 >> [v2:172.21.15.80:6800/1174143345,v1:172.21.15.80:6801/1174143345] conn(0x7f94b002fec0 0x7f94b0032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 >> 172.21.15.80:0/3024866621 conn(0x7f94c80f6530 msgr2=0x7f94c80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 shutdown_connections 2024-02-13T03:26:42.379 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:42.377+0000 7f94d03b3700 1 -- 172.21.15.80:0/3024866621 wait complete. 2024-02-13T03:26:43.019 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying grafana service with default placement... 2024-02-13T03:26:43.019 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-13T03:26:43.559 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f0091534700 1 Processor -- start 2024-02-13T03:26:43.559 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f0091534700 1 -- start start 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f0091534700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f0091534700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0fc970 con 0x7f008c070420 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f008affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f008affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58408/0 (socket says 172.21.15.80:58408) 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.558+0000 7f008affd700 1 -- 172.21.15.80:0/4148266976 learned_addr learned my addr 172.21.15.80:0/4148266976 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f008affd700 1 -- 172.21.15.80:0/4148266976 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c101470 con 0x7f008c070420 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f008affd700 1 --2- 172.21.15.80:0/4148266976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f007c00b5f0 tx=0x7f007c006ee0).ready entity=mon.0 client_cookie=498126d456350746 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0089ffb700 1 -- 172.21.15.80:0/4148266976 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f007c00d600 con 0x7f008c070420 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0089ffb700 1 -- 172.21.15.80:0/4148266976 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f007c00dbf0 con 0x7f008c070420 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0089ffb700 1 -- 172.21.15.80:0/4148266976 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f007c011cb0 con 0x7f008c070420 2024-02-13T03:26:43.561 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 -- 172.21.15.80:0/4148266976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 msgr2=0x7f008c100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:43.562 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 --2- 172.21.15.80:0/4148266976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f007c00b5f0 tx=0x7f007c006ee0).stop 2024-02-13T03:26:43.562 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 -- 172.21.15.80:0/4148266976 shutdown_connections 2024-02-13T03:26:43.562 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 --2- 172.21.15.80:0/4148266976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c100f80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:43.562 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 -- 172.21.15.80:0/4148266976 >> 172.21.15.80:0/4148266976 conn(0x7f008c0f65c0 msgr2=0x7f008c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:43.562 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.559+0000 7f0091534700 1 -- 172.21.15.80:0/4148266976 shutdown_connections 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.560+0000 7f0091534700 1 -- 172.21.15.80:0/4148266976 wait complete. 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.560+0000 7f0091534700 1 Processor -- start 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.560+0000 7f0091534700 1 -- start start 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f0091534700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f0091534700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c10d2b0 con 0x7f008c070420 2024-02-13T03:26:43.564 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f008affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f008affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58424/0 (socket says 172.21.15.80:58424) 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f008affd700 1 -- 172.21.15.80:0/262599921 learned_addr learned my addr 172.21.15.80:0/262599921 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f008affd700 1 -- 172.21.15.80:0/262599921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c00b200 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f008affd700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f007c006800 tx=0x7f007c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f007c01b570 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c10d4b0 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f007c01bb60 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c10e0f0 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.561+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f007c01aa10 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.562+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f007c00f070 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.562+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c10d940 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.562+0000 7f00737fe700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 0x7f00780323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.563+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f007c0415f0 con 0x7f008c070420 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.563+0000 7f008a7fc700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 0x7f00780323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:43.565 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.563+0000 7f008a7fc700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 0x7f00780323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f00800096a0 tx=0x7f0080006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:43.568 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.567+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f007c025980 con 0x7f008c070420 2024-02-13T03:26:43.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:42.373175+0000 mon.smithi080 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi080.vyzlfg(active, since 1.63094s) 2024-02-13T03:26:43.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:42.373974+0000 mgr.smithi080.vyzlfg (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-13T03:26:43.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:42.375976+0000 mon.smithi080 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:43.744 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.743+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 --> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f008c102230 con 0x7f007802ff20 2024-02-13T03:26:43.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.749+0000 7f00737fe700 1 -- 172.21.15.80:0/262599921 <== mgr.14142 v2:172.21.15.80:6800/4099640074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f00800092a0 con 0x7f007802ff20 2024-02-13T03:26:43.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 msgr2=0x7f00780323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 0x7f00780323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f00800096a0 tx=0x7f0080006b40).stop 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 msgr2=0x7f008c10cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f007c006800 tx=0x7f007c006ee0).stop 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 shutdown_connections 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f007802ff20 0x7f00780323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 --2- 172.21.15.80:0/262599921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f008c070420 0x7f008c10cdc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 >> 172.21.15.80:0/262599921 conn(0x7f008c0f65c0 msgr2=0x7f008c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 shutdown_connections 2024-02-13T03:26:43.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:43.751+0000 7f0091534700 1 -- 172.21.15.80:0/262599921 wait complete. 2024-02-13T03:26:44.318 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying node-exporter service with default placement... 2024-02-13T03:26:44.318 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-13T03:26:44.574 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:43.379479+0000 mon.smithi080 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi080.vyzlfg(active, since 2s) 2024-02-13T03:26:44.575 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:43.745296+0000 mgr.smithi080.vyzlfg (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:44.575 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:43.745803+0000 mgr.smithi080.vyzlfg (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-13T03:26:44.575 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:43.750689+0000 mon.smithi080 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:44.879 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.878+0000 7f13d84e2700 1 Processor -- start 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.878+0000 7f13d84e2700 1 -- start start 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d84e2700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d84e2700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00fefc0 con 0x7f13d00703b0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d627e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d627e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58426/0 (socket says 172.21.15.80:58426) 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d627e700 1 -- 172.21.15.80:0/3304498720 learned_addr learned my addr 172.21.15.80:0/3304498720 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d627e700 1 -- 172.21.15.80:0/3304498720 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00ff100 con 0x7f13d00703b0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d627e700 1 --2- 172.21.15.80:0/3304498720 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f13c000b5f0 tx=0x7f13c0006ee0).ready entity=mon.0 client_cookie=b91fbbb5c03d2fb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d527c700 1 -- 172.21.15.80:0/3304498720 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c000d600 con 0x7f13d00703b0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d527c700 1 -- 172.21.15.80:0/3304498720 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f13c000dbf0 con 0x7f13d00703b0 2024-02-13T03:26:44.881 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.879+0000 7f13d527c700 1 -- 172.21.15.80:0/3304498720 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c0011cb0 con 0x7f13d00703b0 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- 172.21.15.80:0/3304498720 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 msgr2=0x7f13d00fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 --2- 172.21.15.80:0/3304498720 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f13c000b5f0 tx=0x7f13c0006ee0).stop 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- 172.21.15.80:0/3304498720 shutdown_connections 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 --2- 172.21.15.80:0/3304498720 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fead0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- 172.21.15.80:0/3304498720 >> 172.21.15.80:0/3304498720 conn(0x7f13d00f6530 msgr2=0x7f13d00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:44.884 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- 172.21.15.80:0/3304498720 shutdown_connections 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- 172.21.15.80:0/3304498720 wait complete. 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 Processor -- start 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.880+0000 7f13d84e2700 1 -- start start 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d84e2700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d84e2700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00fa9d0 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d627e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d627e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58440/0 (socket says 172.21.15.80:58440) 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d627e700 1 -- 172.21.15.80:0/945323117 learned_addr learned my addr 172.21.15.80:0/945323117 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d627e700 1 -- 172.21.15.80:0/945323117 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c000b200 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d627e700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f13c0006800 tx=0x7f13c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c001b570 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f13c001bb60 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d00fab70 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c001a9f0 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.881+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d00fb000 con 0x7f13d00703b0 2024-02-13T03:26:44.885 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.882+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f13c000f070 con 0x7f13d00703b0 2024-02-13T03:26:44.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.882+0000 7f13beffd700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 0x7f13b80323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:44.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.882+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d0045ad0 con 0x7f13d00703b0 2024-02-13T03:26:44.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.882+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f13c00415c0 con 0x7f13d00703b0 2024-02-13T03:26:44.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.882+0000 7f13d5a7d700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 0x7f13b80323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:44.886 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.883+0000 7f13d5a7d700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 0x7f13b80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f13cc0096a0 tx=0x7f13cc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:44.887 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:44.886+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13c0016070 con 0x7f13d00703b0 2024-02-13T03:26:45.061 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.060+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 --> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d00fffc0 con 0x7f13b802ff20 2024-02-13T03:26:45.064 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.064+0000 7f13beffd700 1 -- 172.21.15.80:0/945323117 <== mgr.14142 v2:172.21.15.80:6800/4099640074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f13d00fffc0 con 0x7f13b802ff20 2024-02-13T03:26:45.064 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.065+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 msgr2=0x7f13b80323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.065+0000 7f13d84e2700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 0x7f13b80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f13cc0096a0 tx=0x7f13cc006b40).stop 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.065+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 msgr2=0x7f13d00fc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.065+0000 7f13d84e2700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f13c0006800 tx=0x7f13c0006ee0).stop 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.065+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 shutdown_connections 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.066+0000 7f13d84e2700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f13b802ff20 0x7f13b80323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.066+0000 7f13d84e2700 1 --2- 172.21.15.80:0/945323117 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13d00703b0 0x7f13d00fc550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:45.067 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.066+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 >> 172.21.15.80:0/945323117 conn(0x7f13d00f6530 msgr2=0x7f13d00f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:45.068 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.066+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 shutdown_connections 2024-02-13T03:26:45.068 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:45.066+0000 7f13d84e2700 1 -- 172.21.15.80:0/945323117 wait complete. 2024-02-13T03:26:45.616 INFO:teuthology.orchestra.run.smithi080.stderr:Deploying alertmanager service with default placement... 2024-02-13T03:26:45.617 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-13T03:26:46.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:45.062032+0000 mgr.smithi080.vyzlfg (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:46.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:45.062665+0000 mgr.smithi080.vyzlfg (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-13T03:26:46.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:45.064843+0000 mon.smithi080 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.231+0000 7f7ff622c700 1 Processor -- start 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.231+0000 7f7ff622c700 1 -- start start 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.231+0000 7f7ff622c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.231+0000 7f7ff622c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff00fef80 con 0x7f7ff00703f0 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fef7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fef7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58456/0 (socket says 172.21.15.80:58456) 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fef7fe700 1 -- 172.21.15.80:0/296231456 learned_addr learned my addr 172.21.15.80:0/296231456 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fef7fe700 1 -- 172.21.15.80:0/296231456 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff00ff0c0 con 0x7f7ff00703f0 2024-02-13T03:26:46.749 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fef7fe700 1 --2- 172.21.15.80:0/296231456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7fe400b5f0 tx=0x7f7fe4006ee0).ready entity=mon.0 client_cookie=3c97f70d475a13db server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fee7fc700 1 -- 172.21.15.80:0/296231456 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fe400d600 con 0x7f7ff00703f0 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fee7fc700 1 -- 172.21.15.80:0/296231456 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7fe400dbf0 con 0x7f7ff00703f0 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7fee7fc700 1 -- 172.21.15.80:0/296231456 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fe4011ce0 con 0x7f7ff00703f0 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7ff622c700 1 -- 172.21.15.80:0/296231456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 msgr2=0x7f7ff00fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7ff622c700 1 --2- 172.21.15.80:0/296231456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7fe400b5f0 tx=0x7f7fe4006ee0).stop 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7ff622c700 1 -- 172.21.15.80:0/296231456 shutdown_connections 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.232+0000 7f7ff622c700 1 --2- 172.21.15.80:0/296231456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff00fea90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 -- 172.21.15.80:0/296231456 >> 172.21.15.80:0/296231456 conn(0x7f7ff00f6530 msgr2=0x7f7ff00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 -- 172.21.15.80:0/296231456 shutdown_connections 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 -- 172.21.15.80:0/296231456 wait complete. 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 Processor -- start 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 -- start start 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:46.750 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7ff622c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff010b040 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7fef7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7fef7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58460/0 (socket says 172.21.15.80:58460) 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.233+0000 7f7fef7fe700 1 -- 172.21.15.80:0/3830828175 learned_addr learned my addr 172.21.15.80:0/3830828175 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7fef7fe700 1 -- 172.21.15.80:0/3830828175 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe400b200 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7fef7fe700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f7fe4011ed0 tx=0x7f7fe4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fe401b570 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7fe401bb60 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff010b240 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fe401aad0 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.234+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff010b6d0 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7fe400f070 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7fd7fff700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 0x7f7fdc0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff0045ad0 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7fe4041510 con 0x7f7ff00703f0 2024-02-13T03:26:46.751 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7feeffd700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 0x7f7fdc0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.235+0000 7f7feeffd700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 0x7f7fdc0323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7fe00096a0 tx=0x7f7fe0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.239+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fe4016070 con 0x7f7ff00703f0 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.415+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 --> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff010bf90 con 0x7f7fdc02ff20 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.418+0000 7f7fd7fff700 1 -- 172.21.15.80:0/3830828175 <== mgr.14142 v2:172.21.15.80:6800/4099640074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f7ff010bf90 con 0x7f7fdc02ff20 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 msgr2=0x7f7fdc0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 0x7f7fdc0323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7fe00096a0 tx=0x7f7fe0006b40).stop 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 msgr2=0x7f7ff010ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f7fe4011ed0 tx=0x7f7fe4006ee0).stop 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 shutdown_connections 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f7fdc02ff20 0x7f7fdc0323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 --2- 172.21.15.80:0/3830828175 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ff00703f0 0x7f7ff010ab50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 >> 172.21.15.80:0/3830828175 conn(0x7f7ff00f6530 msgr2=0x7f7ff00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:46.752 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 shutdown_connections 2024-02-13T03:26:46.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:46.420+0000 7f7ff622c700 1 -- 172.21.15.80:0/3830828175 wait complete. 2024-02-13T03:26:46.753 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-13T03:26:47.300 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-13T03:26:47.571 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:46.416850+0000 mgr.smithi080.vyzlfg (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:26:47.572 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:26:46.417375+0000 mgr.smithi080.vyzlfg (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-13T03:26:47.572 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:46.419562+0000 mon.smithi080 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:47.837 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.835+0000 7f205137f700 1 Processor -- start 2024-02-13T03:26:47.837 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.835+0000 7f205137f700 1 -- start start 2024-02-13T03:26:47.837 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.835+0000 7f205137f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:47.837 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f205137f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0feff0 con 0x7f204c0704a0 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f204affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f204affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58464/0 (socket says 172.21.15.80:58464) 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f204affd700 1 -- 172.21.15.80:0/3653646163 learned_addr learned my addr 172.21.15.80:0/3653646163 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f204affd700 1 -- 172.21.15.80:0/3653646163 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0ff130 con 0x7f204c0704a0 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f204affd700 1 --2- 172.21.15.80:0/3653646163 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f204000b5f0 tx=0x7f2040006ee0).ready entity=mon.0 client_cookie=b708c99667b511f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:47.838 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f2049ffb700 1 -- 172.21.15.80:0/3653646163 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f204000d600 con 0x7f204c0704a0 2024-02-13T03:26:47.841 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f2049ffb700 1 -- 172.21.15.80:0/3653646163 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f204000dbf0 con 0x7f204c0704a0 2024-02-13T03:26:47.841 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.836+0000 7f2049ffb700 1 -- 172.21.15.80:0/3653646163 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2040011cb0 con 0x7f204c0704a0 2024-02-13T03:26:47.841 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 -- 172.21.15.80:0/3653646163 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 msgr2=0x7f204c0feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:47.841 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 --2- 172.21.15.80:0/3653646163 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f204000b5f0 tx=0x7f2040006ee0).stop 2024-02-13T03:26:47.841 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 -- 172.21.15.80:0/3653646163 shutdown_connections 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 --2- 172.21.15.80:0/3653646163 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0feb00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 -- 172.21.15.80:0/3653646163 >> 172.21.15.80:0/3653646163 conn(0x7f204c0f65c0 msgr2=0x7f204c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 -- 172.21.15.80:0/3653646163 shutdown_connections 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.837+0000 7f205137f700 1 -- 172.21.15.80:0/3653646163 wait complete. 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f205137f700 1 Processor -- start 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f205137f700 1 -- start start 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f205137f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f205137f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0faa00 con 0x7f204c0704a0 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f204affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:47.842 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f204affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58472/0 (socket says 172.21.15.80:58472) 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.838+0000 7f204affd700 1 -- 172.21.15.80:0/1613991938 learned_addr learned my addr 172.21.15.80:0/1613991938 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f204affd700 1 -- 172.21.15.80:0/1613991938 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204000b200 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f204affd700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2040006800 tx=0x7f2040006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f204001b570 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c0faba0 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f204001bb60 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f204001a9f0 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.839+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c0fb030 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.840+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c045ad0 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.840+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f204000f070 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.840+0000 7f20337fe700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 0x7f20380323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.840+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2040041ce0 con 0x7f204c0704a0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.840+0000 7f204a7fc700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 0x7f20380323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:47.843 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.841+0000 7f204a7fc700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 0x7f20380323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f203c0096a0 tx=0x7f203c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:47.845 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.844+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2040023b50 con 0x7f204c0704a0 2024-02-13T03:26:47.993 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.992+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f204c0ffe00 con 0x7f204c0704a0 2024-02-13T03:26:47.995 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.995+0000 7f20337fe700 1 -- 172.21.15.80:0/1613991938 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2040016030 con 0x7f204c0704a0 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 msgr2=0x7f20380323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 0x7f20380323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f203c0096a0 tx=0x7f203c006b40).stop 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 msgr2=0x7f204c0fc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2040006800 tx=0x7f2040006ee0).stop 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 shutdown_connections 2024-02-13T03:26:47.998 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f203802ff20 0x7f20380323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:47.999 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 --2- 172.21.15.80:0/1613991938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f204c0704a0 0x7f204c0fc580 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:47.999 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 >> 172.21.15.80:0/1613991938 conn(0x7f204c0f65c0 msgr2=0x7f204c0f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:47.999 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 shutdown_connections 2024-02-13T03:26:47.999 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:47.997+0000 7f205137f700 1 -- 172.21.15.80:0/1613991938 wait complete. 2024-02-13T03:26:48.724 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-13T03:26:49.264 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:47.996063+0000 mon.smithi080 (mon.0) 82 : audit [INF] from='client.? 172.21.15.80:0/1613991938' entity='client.admin' 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188c5eb700 1 Processor -- start 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188c5eb700 1 -- start start 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188c5eb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188c5eb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840fb6a0 con 0x7f18840fad90 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188a387700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188a387700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44518/0 (socket says 172.21.15.80:44518) 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.498+0000 7f188a387700 1 -- 172.21.15.80:0/1719478691 learned_addr learned my addr 172.21.15.80:0/1719478691 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f188a387700 1 -- 172.21.15.80:0/1719478691 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18840fb7e0 con 0x7f18840fad90 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f188a387700 1 --2- 172.21.15.80:0/1719478691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f188000b5f0 tx=0x7f1880006ee0).ready entity=mon.0 client_cookie=b0a30ab67d9f3ea1 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f1889385700 1 -- 172.21.15.80:0/1719478691 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188000d600 con 0x7f18840fad90 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f1889385700 1 -- 172.21.15.80:0/1719478691 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f188000dbf0 con 0x7f18840fad90 2024-02-13T03:26:49.687 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f1889385700 1 -- 172.21.15.80:0/1719478691 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1880011cb0 con 0x7f18840fad90 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f188c5eb700 1 -- 172.21.15.80:0/1719478691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 msgr2=0x7f18840fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.499+0000 7f188c5eb700 1 --2- 172.21.15.80:0/1719478691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f188000b5f0 tx=0x7f1880006ee0).stop 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 -- 172.21.15.80:0/1719478691 shutdown_connections 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 --2- 172.21.15.80:0/1719478691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f18840fb1b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 -- 172.21.15.80:0/1719478691 >> 172.21.15.80:0/1719478691 conn(0x7f18840f6530 msgr2=0x7f18840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 -- 172.21.15.80:0/1719478691 shutdown_connections 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 -- 172.21.15.80:0/1719478691 wait complete. 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 Processor -- start 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.500+0000 7f188c5eb700 1 -- start start 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188c5eb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188c5eb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884108df0 con 0x7f18840fad90 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188a387700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188a387700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44528/0 (socket says 172.21.15.80:44528) 2024-02-13T03:26:49.688 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188a387700 1 -- 172.21.15.80:0/3295145381 learned_addr learned my addr 172.21.15.80:0/3295145381 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188a387700 1 -- 172.21.15.80:0/3295145381 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188000b200 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188a387700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f1880011ea0 tx=0x7f1880006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188001b570 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f188001bb60 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1884108ff0 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188001a8d0 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.501+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1884109480 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f188000f070 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f1872ffd700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 0x7f186c034680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1880041540 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1884045ad0 con 0x7f18840fad90 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f1889b86700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 0x7f186c034680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.502+0000 7f1889b86700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 0x7f186c034680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f18740096a0 tx=0x7f1874006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:49.689 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.510+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1880016070 con 0x7f18840fad90 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.665+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f18840fcae0 con 0x7f18840fad90 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.668+0000 7f1872ffd700 1 -- 172.21.15.80:0/3295145381 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1880022d20 con 0x7f18840fad90 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 msgr2=0x7f186c034680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 0x7f186c034680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f18740096a0 tx=0x7f1874006b40).stop 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 msgr2=0x7f1884108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f1880011ea0 tx=0x7f1880006ee0).stop 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 shutdown_connections 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f186c0321c0 0x7f186c034680 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 --2- 172.21.15.80:0/3295145381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18840fad90 0x7f1884108900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 >> 172.21.15.80:0/3295145381 conn(0x7f18840f6530 msgr2=0x7f18840ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 shutdown_connections 2024-02-13T03:26:49.690 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:49.670+0000 7f188c5eb700 1 -- 172.21.15.80:0/3295145381 wait complete. 2024-02-13T03:26:50.505 INFO:teuthology.orchestra.run.smithi080.stderr:Enabling the dashboard module... 2024-02-13T03:26:50.505 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-13T03:26:50.773 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:49.668292+0000 mon.smithi080 (mon.0) 83 : audit [INF] from='client.? 172.21.15.80:0/3295145381' entity='client.admin' 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.088+0000 7f478dd4f700 1 Processor -- start 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.088+0000 7f478dd4f700 1 -- start start 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478dd4f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478dd4f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880fb440 con 0x7f47880fab30 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478cd4d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478cd4d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44532/0 (socket says 172.21.15.80:44532) 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478cd4d700 1 -- 172.21.15.80:0/1952232736 learned_addr learned my addr 172.21.15.80:0/1952232736 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478cd4d700 1 -- 172.21.15.80:0/1952232736 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880fb580 con 0x7f47880fab30 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f478cd4d700 1 --2- 172.21.15.80:0/1952232736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f477c00b5f0 tx=0x7f477c006ee0).ready entity=mon.0 client_cookie=45a3b0abe4ba221a server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:51.092 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f47877fe700 1 -- 172.21.15.80:0/1952232736 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c00d600 con 0x7f47880fab30 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f47877fe700 1 -- 172.21.15.80:0/1952232736 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f477c00dbf0 con 0x7f47880fab30 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.089+0000 7f47877fe700 1 -- 172.21.15.80:0/1952232736 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c011cb0 con 0x7f47880fab30 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- 172.21.15.80:0/1952232736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 msgr2=0x7f47880faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1952232736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f477c00b5f0 tx=0x7f477c006ee0).stop 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- 172.21.15.80:0/1952232736 shutdown_connections 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1952232736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f47880faf50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- 172.21.15.80:0/1952232736 >> 172.21.15.80:0/1952232736 conn(0x7f47880f6390 msgr2=0x7f47880f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- 172.21.15.80:0/1952232736 shutdown_connections 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- 172.21.15.80:0/1952232736 wait complete. 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 Processor -- start 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.090+0000 7f478dd4f700 1 -- start start 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478dd4f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478dd4f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478810ae60 con 0x7f47880fab30 2024-02-13T03:26:51.093 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478cd4d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478cd4d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44542/0 (socket says 172.21.15.80:44542) 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478cd4d700 1 -- 172.21.15.80:0/1187154989 learned_addr learned my addr 172.21.15.80:0/1187154989 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478cd4d700 1 -- 172.21.15.80:0/1187154989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c00b200 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478cd4d700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f477c011ea0 tx=0x7f477c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c01b570 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f477c01bb60 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c01a8d0 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.091+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478810b060 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.092+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478810b4f0 con 0x7f47880fab30 2024-02-13T03:26:51.094 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.092+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f477c00f070 con 0x7f47880fab30 2024-02-13T03:26:51.095 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.092+0000 7f47857fa700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 0x7f476c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:51.095 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.092+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f477c041640 con 0x7f47880fab30 2024-02-13T03:26:51.095 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.092+0000 7f4787fff700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 0x7f476c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:51.095 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.093+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4788045ad0 con 0x7f47880fab30 2024-02-13T03:26:51.095 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.093+0000 7f4787fff700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 0x7f476c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f47740096a0 tx=0x7f4774006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:51.097 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.097+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f477c016070 con 0x7f47880fab30 2024-02-13T03:26:51.286 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.286+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4788104a80 con 0x7f47880fab30 2024-02-13T03:26:51.415 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.416+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 7 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f477c01b8b0 con 0x7f47880fab30 2024-02-13T03:26:51.674 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.674+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 8 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f477c01ac10 con 0x7f47880fab30 2024-02-13T03:26:51.675 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.675+0000 7f47857fa700 1 -- 172.21.15.80:0/1187154989 <== mon.0 v2:172.21.15.80:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f477c02ea30 con 0x7f47880fab30 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 msgr2=0x7f476c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 0x7f476c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f47740096a0 tx=0x7f4774006b40).stop 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 msgr2=0x7f478810a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f477c011ea0 tx=0x7f477c006ee0).stop 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 shutdown_connections 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f476c02ff20 0x7f476c0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 --2- 172.21.15.80:0/1187154989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47880fab30 0x7f478810a970 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 >> 172.21.15.80:0/1187154989 conn(0x7f47880f6390 msgr2=0x7f4788070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 shutdown_connections 2024-02-13T03:26:51.678 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:51.677+0000 7f478dd4f700 1 -- 172.21.15.80:0/1187154989 wait complete. 2024-02-13T03:26:51.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:51.287270+0000 mon.smithi080 (mon.0) 84 : audit [INF] from='client.? 172.21.15.80:0/1187154989' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-13T03:26:51.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:51.416188+0000 mon.smithi080 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:51.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:51.632335+0000 mon.smithi080 (mon.0) 86 : audit [INF] from='mgr.14142 172.21.15.80:0/4222552803' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:52.315 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-13T03:26:52.925 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:51.675348+0000 mon.smithi080 (mon.0) 87 : audit [INF] from='client.? 172.21.15.80:0/1187154989' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-13T03:26:52.925 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:51.675409+0000 mon.smithi080 (mon.0) 88 : cluster [DBG] mgrmap e13: smithi080.vyzlfg(active, since 10s) 2024-02-13T03:26:53.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.065+0000 7f0a9aaa9700 1 Processor -- start 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a9aaa9700 1 -- start start 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a9aaa9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a9aaa9700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940fedc0 con 0x7f0a940703f0 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a99aa7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a99aa7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44578/0 (socket says 172.21.15.80:44578) 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a99aa7700 1 -- 172.21.15.80:0/2071705995 learned_addr learned my addr 172.21.15.80:0/2071705995 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a99aa7700 1 -- 172.21.15.80:0/2071705995 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940fef00 con 0x7f0a940703f0 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a99aa7700 1 --2- 172.21.15.80:0/2071705995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f0a9000b5f0 tx=0x7f0a90006ee0).ready entity=mon.0 client_cookie=75265ead7f8be710 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.066+0000 7f0a98aa5700 1 -- 172.21.15.80:0/2071705995 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9000d600 con 0x7f0a940703f0 2024-02-13T03:26:53.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a98aa5700 1 -- 172.21.15.80:0/2071705995 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a9000dc30 con 0x7f0a940703f0 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a98aa5700 1 -- 172.21.15.80:0/2071705995 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a90011d60 con 0x7f0a940703f0 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2071705995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 msgr2=0x7f0a940fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2071705995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f0a9000b5f0 tx=0x7f0a90006ee0).stop 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2071705995 shutdown_connections 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2071705995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a940fe8d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2071705995 >> 172.21.15.80:0/2071705995 conn(0x7f0a940f6390 msgr2=0x7f0a940f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2071705995 shutdown_connections 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.067+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2071705995 wait complete. 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a9aaa9700 1 Processor -- start 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a9aaa9700 1 -- start start 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a9aaa9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a9aaa9700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9410af30 con 0x7f0a940703f0 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a99aa7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a99aa7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44584/0 (socket says 172.21.15.80:44584) 2024-02-13T03:26:53.181 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a99aa7700 1 -- 172.21.15.80:0/2865057272 learned_addr learned my addr 172.21.15.80:0/2865057272 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a99aa7700 1 -- 172.21.15.80:0/2865057272 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9000b200 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.068+0000 7f0a99aa7700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0a9000e040 tx=0x7f0a90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a90011e10 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a9001b8e0 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9001a930 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9410b130 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9410b5c0 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.069+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0a9000f070 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.070+0000 7f0a827fc700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 0x7f0a7c032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.070+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0a90040ed0 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.070+0000 7f0a992a6700 1 -- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 msgr2=0x7f0a7c032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.070+0000 7f0a992a6700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 0x7f0a7c032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.070+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94045ad0 con 0x7f0a940703f0 2024-02-13T03:26:53.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.074+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a90016070 con 0x7f0a940703f0 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.269+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0a941049e0 con 0x7f0a940703f0 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.270+0000 7f0a827fc700 1 -- 172.21.15.80:0/2865057272 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f0a900107d0 con 0x7f0a940703f0 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "available": true, 2024-02-13T03:26:53.271 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "active_name": "smithi080.vyzlfg", 2024-02-13T03:26:53.272 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-13T03:26:53.272 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:53.272 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.271+0000 7f0a992a6700 1 -- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 msgr2=0x7f0a7c032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:53.272 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.271+0000 7f0a992a6700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 0x7f0a7c032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.271+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 msgr2=0x7f0a7c032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.271+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 0x7f0a7c032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.271+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 msgr2=0x7f0a9410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0a9000e040 tx=0x7f0a90006ee0).stop 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 shutdown_connections 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f0a7c02ffc0 0x7f0a7c032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 --2- 172.21.15.80:0/2865057272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a940703f0 0x7f0a9410aa40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 >> 172.21.15.80:0/2865057272 conn(0x7f0a940f6390 msgr2=0x7f0a940f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:53.273 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 shutdown_connections 2024-02-13T03:26:53.274 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:53.272+0000 7f0a9aaa9700 1 -- 172.21.15.80:0/2865057272 wait complete. 2024-02-13T03:26:53.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:53 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:53.271266+0000 mon.smithi080 (mon.0) 89 : audit [DBG] from='client.? 172.21.15.80:0/2865057272' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T03:26:54.050 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for the mgr to restart... 2024-02-13T03:26:54.050 INFO:teuthology.orchestra.run.smithi080.stderr:Waiting for mgr epoch 13... 2024-02-13T03:26:54.050 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-13T03:26:54.629 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.626+0000 7f48c5272700 1 Processor -- start 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.626+0000 7f48c5272700 1 -- start start 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.626+0000 7f48c5272700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.626+0000 7f48c5272700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c00fc980 con 0x7f48c00703b0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.626+0000 7f48beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44592/0 (socket says 172.21.15.80:44592) 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48beffd700 1 -- 172.21.15.80:0/3412973477 learned_addr learned my addr 172.21.15.80:0/3412973477 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48beffd700 1 -- 172.21.15.80:0/3412973477 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0101480 con 0x7f48c00703b0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48beffd700 1 --2- 172.21.15.80:0/3412973477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f48b000b5f0 tx=0x7f48b0006ee0).ready entity=mon.0 client_cookie=2af2d3432c4d09b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48bdffb700 1 -- 172.21.15.80:0/3412973477 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b000d600 con 0x7f48c00703b0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48bdffb700 1 -- 172.21.15.80:0/3412973477 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48b000dc30 con 0x7f48c00703b0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.627+0000 7f48bdffb700 1 -- 172.21.15.80:0/3412973477 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0011d30 con 0x7f48c00703b0 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 -- 172.21.15.80:0/3412973477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 msgr2=0x7f48c0100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:54.630 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 --2- 172.21.15.80:0/3412973477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f48b000b5f0 tx=0x7f48b0006ee0).stop 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 -- 172.21.15.80:0/3412973477 shutdown_connections 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 --2- 172.21.15.80:0/3412973477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c0100f90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 -- 172.21.15.80:0/3412973477 >> 172.21.15.80:0/3412973477 conn(0x7f48c00f65b0 msgr2=0x7f48c00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 -- 172.21.15.80:0/3412973477 shutdown_connections 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.628+0000 7f48c5272700 1 -- 172.21.15.80:0/3412973477 wait complete. 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48c5272700 1 Processor -- start 2024-02-13T03:26:54.631 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48c5272700 1 -- start start 2024-02-13T03:26:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48c5272700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48c5272700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c010d3b0 con 0x7f48c00703b0 2024-02-13T03:26:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:44600/0 (socket says 172.21.15.80:44600) 2024-02-13T03:26:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.629+0000 7f48beffd700 1 -- 172.21.15.80:0/3049489674 learned_addr learned my addr 172.21.15.80:0/3049489674 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48beffd700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b000b200 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48beffd700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f48b0006800 tx=0x7f48b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b000d960 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48b000dac0 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b001a820 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c010d5b0 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.630+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c010e160 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f48b000f070 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48a77fe700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f48b0042350 con 0x7f48c00703b0 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48be7fc700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 msgr2=0x7f48ac032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-13T03:26:54.633 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.631+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48c0059100 con 0x7f48ac02ffc0 2024-02-13T03:26:54.833 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.832+0000 7f48be7fc700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 msgr2=0x7f48ac032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:54.833 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:54.832+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-13T03:26:55.234 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:55.233+0000 7f48be7fc700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 msgr2=0x7f48ac032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:55.234 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:55.233+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-13T03:26:56.035 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:56.034+0000 7f48be7fc700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 msgr2=0x7f48ac032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.80:6800/4099640074 2024-02-13T03:26:56.035 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:56.034+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-13T03:26:56.719 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:56.718+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f48b001bce0 con 0x7f48c00703b0 2024-02-13T03:26:56.719 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:56.718+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 msgr2=0x7f48ac032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-13T03:26:56.719 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:56.718+0000 7f48a77fe700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:57.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:56.667139+0000 mon.smithi080 (mon.0) 90 : cluster [INF] Active manager daemon smithi080.vyzlfg restarted 2024-02-13T03:26:57.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:56.667574+0000 mon.smithi080 (mon.0) 91 : cluster [INF] Activating manager daemon smithi080.vyzlfg 2024-02-13T03:26:57.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:56.669260+0000 mon.smithi080 (mon.0) 92 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-13T03:26:57.726 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.724+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f48b00257e0 con 0x7f48c00703b0 2024-02-13T03:26:57.726 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.724+0000 7f48a77fe700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 0x7f48ac035e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:57.727 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.724+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48c0059100 con 0x7f48ac033aa0 2024-02-13T03:26:57.727 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.725+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 0x7f48ac035e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:57.727 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.725+0000 7f48be7fc700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 0x7f48ac035e90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f48b40036d0 tx=0x7f48b4006d20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:57.727 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.726+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f48c0059100 con 0x7f48ac033aa0 2024-02-13T03:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.731+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f48c010da40 con 0x7f48ac033aa0 2024-02-13T03:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48a77fe700 1 -- 172.21.15.80:0/3049489674 <== mgr.14162 v2:172.21.15.80:6800/2233275020 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f48c010da40 con 0x7f48ac033aa0 2024-02-13T03:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: { 2024-02-13T03:26:57.733 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: "initialized": true 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: } 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 msgr2=0x7f48ac035e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 0x7f48ac035e90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f48b40036d0 tx=0x7f48b4006d20).stop 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 msgr2=0x7f48c010cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f48b0006800 tx=0x7f48b0006ee0).stop 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 shutdown_connections 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48ac033aa0 0x7f48ac035e90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48c00703b0 0x7f48c010cec0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 --2- 172.21.15.80:0/3049489674 >> [v2:172.21.15.80:6800/4099640074,v1:172.21.15.80:6801/4099640074] conn(0x7f48ac02ffc0 0x7f48ac032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 >> 172.21.15.80:0/3049489674 conn(0x7f48c00f65b0 msgr2=0x7f48c00f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 shutdown_connections 2024-02-13T03:26:57.734 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:57.732+0000 7f48c5272700 1 -- 172.21.15.80:0/3049489674 wait complete. 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:56.719973+0000 mon.smithi080 (mon.0) 93 : cluster [DBG] mgrmap e14: smithi080.vyzlfg(active, starting, since 0.0524852s) 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.721046+0000 mon.smithi080 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata", "id": "smithi080"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.723105+0000 mon.smithi080 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mgr metadata", "who": "smithi080.vyzlfg", "id": "smithi080.vyzlfg"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.723269+0000 mon.smithi080 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.723388+0000 mon.smithi080 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.723501+0000 mon.smithi080 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:56.737375+0000 mon.smithi080 (mon.0) 99 : cluster [INF] Manager daemon smithi080.vyzlfg is now available 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.833433+0000 mon.smithi080 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.835647+0000 mon.smithi080 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.851635+0000 mon.smithi080 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:26:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:56.859625+0000 mon.smithi080 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:26:58.560 INFO:teuthology.orchestra.run.smithi080.stderr:mgr epoch 13 is available 2024-02-13T03:26:58.560 INFO:teuthology.orchestra.run.smithi080.stderr:Generating a dashboard self-signed certificate... 2024-02-13T03:26:58.560 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-13T03:26:58.825 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:57.726749+0000 mgr.smithi080.vyzlfg (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T03:26:58.825 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:57.727521+0000 mon.smithi080 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi080.vyzlfg(active, since 1.06003s) 2024-02-13T03:26:58.825 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:57.733181+0000 mgr.smithi080.vyzlfg (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T03:26:58.825 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:26:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:58.256993+0000 mon.smithi080 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:26:59.254 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f981de29700 1 Processor -- start 2024-02-13T03:26:59.254 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f981de29700 1 -- start start 2024-02-13T03:26:59.254 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f981de29700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:59.254 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f981de29700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98180fbd20 con 0x7f98180fb410 2024-02-13T03:26:59.254 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f98177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f98177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41314/0 (socket says 172.21.15.80:41314) 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.251+0000 7f98177fe700 1 -- 172.21.15.80:0/3015101691 learned_addr learned my addr 172.21.15.80:0/3015101691 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f98177fe700 1 -- 172.21.15.80:0/3015101691 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180fbe60 con 0x7f98180fb410 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f98177fe700 1 --2- 172.21.15.80:0/3015101691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f980800b870 tx=0x7f9808006f70).ready entity=mon.0 client_cookie=d2e9cef89c82662a server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f98167fc700 1 -- 172.21.15.80:0/3015101691 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980800dab0 con 0x7f98180fb410 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f98167fc700 1 -- 172.21.15.80:0/3015101691 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f980800dc10 con 0x7f98180fb410 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f98167fc700 1 -- 172.21.15.80:0/3015101691 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980800f660 con 0x7f98180fb410 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f981de29700 1 -- 172.21.15.80:0/3015101691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 msgr2=0x7f98180fb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f981de29700 1 --2- 172.21.15.80:0/3015101691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f980800b870 tx=0x7f9808006f70).stop 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.252+0000 7f981de29700 1 -- 172.21.15.80:0/3015101691 shutdown_connections 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 --2- 172.21.15.80:0/3015101691 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f98180fb830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 -- 172.21.15.80:0/3015101691 >> 172.21.15.80:0/3015101691 conn(0x7f98180f5d20 msgr2=0x7f98180f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 -- 172.21.15.80:0/3015101691 shutdown_connections 2024-02-13T03:26:59.255 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 -- 172.21.15.80:0/3015101691 wait complete. 2024-02-13T03:26:59.256 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 Processor -- start 2024-02-13T03:26:59.256 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 -- start start 2024-02-13T03:26:59.256 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:59.256 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.253+0000 7f981de29700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9818108e20 con 0x7f98180fb410 2024-02-13T03:26:59.257 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f98177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:59.257 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f98177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41320/0 (socket says 172.21.15.80:41320) 2024-02-13T03:26:59.257 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f98177fe700 1 -- 172.21.15.80:0/359991002 learned_addr learned my addr 172.21.15.80:0/359991002 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:26:59.257 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f98177fe700 1 -- 172.21.15.80:0/359991002 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980800b480 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f98177fe700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9808006800 tx=0x7f9808006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.254+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980800daa0 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.255+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9818109020 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.255+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f980800fdb0 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.255+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9808019de0 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.255+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98181094b0 con 0x7f98180fb410 2024-02-13T03:26:59.258 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.256+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9818045ad0 con 0x7f98180fb410 2024-02-13T03:26:59.259 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.257+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f980802b060 con 0x7f98180fb410 2024-02-13T03:26:59.259 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.258+0000 7f97fffff700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 0x7f9804032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:26:59.259 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.258+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9808044440 con 0x7f98180fb410 2024-02-13T03:26:59.259 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.258+0000 7f9816ffd700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 0x7f9804032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:26:59.259 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.258+0000 7f9816ffd700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 0x7f9804032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f980c0096a0 tx=0x7f980c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:26:59.262 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.262+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9808019450 con 0x7f98180fb410 2024-02-13T03:26:59.424 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.422+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f9818109d10 con 0x7f980402ff70 2024-02-13T03:26:59.475 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.474+0000 7f97fffff700 1 -- 172.21.15.80:0/359991002 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9818109d10 con 0x7f980402ff70 2024-02-13T03:26:59.475 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-13T03:26:59.477 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 msgr2=0x7f9804032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 0x7f9804032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f980c0096a0 tx=0x7f980c006b40).stop 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 msgr2=0x7f9818108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9808006800 tx=0x7f9808006f70).stop 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 shutdown_connections 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f980402ff70 0x7f9804032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 --2- 172.21.15.80:0/359991002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f98180fb410 0x7f9818108930 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 >> 172.21.15.80:0/359991002 conn(0x7f98180f5d20 msgr2=0x7f98180ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 shutdown_connections 2024-02-13T03:26:59.478 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:26:59.476+0000 7f981de29700 1 -- 172.21.15.80:0/359991002 wait complete. 2024-02-13T03:27:00.421 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:26:59.258443+0000 mon.smithi080 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi080.vyzlfg(active, since 2s) 2024-02-13T03:27:00.421 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:59.424502+0000 mgr.smithi080.vyzlfg (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:27:00.421 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:59.472749+0000 mon.smithi080 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:00.421 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:26:59.475286+0000 mon.smithi080 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:00.774 INFO:teuthology.orchestra.run.smithi080.stderr:Creating initial admin user... 2024-02-13T03:27:00.775 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprkwc0j11:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.320+0000 7fdc8eb0c700 1 Processor -- start 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.320+0000 7fdc8eb0c700 1 -- start start 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.320+0000 7fdc8eb0c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.320+0000 7fdc8eb0c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880fc8c0 con 0x7fdc880703b0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc8c8a8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc8c8a8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41328/0 (socket says 172.21.15.80:41328) 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc8c8a8700 1 -- 172.21.15.80:0/2510230404 learned_addr learned my addr 172.21.15.80:0/2510230404 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc8c8a8700 1 -- 172.21.15.80:0/2510230404 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc881013c0 con 0x7fdc880703b0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc8c8a8700 1 --2- 172.21.15.80:0/2510230404 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc7c00b5f0 tx=0x7fdc7c006ee0).ready entity=mon.0 client_cookie=4f978dcf0f614313 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc877fe700 1 -- 172.21.15.80:0/2510230404 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c00d600 con 0x7fdc880703b0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc877fe700 1 -- 172.21.15.80:0/2510230404 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc7c00dc30 con 0x7fdc880703b0 2024-02-13T03:27:01.323 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.321+0000 7fdc877fe700 1 -- 172.21.15.80:0/2510230404 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c011d30 con 0x7fdc880703b0 2024-02-13T03:27:01.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/2510230404 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 msgr2=0x7fdc88100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:01.324 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/2510230404 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc7c00b5f0 tx=0x7fdc7c006ee0).stop 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/2510230404 shutdown_connections 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/2510230404 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc88100ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/2510230404 >> 172.21.15.80:0/2510230404 conn(0x7fdc880f6530 msgr2=0x7fdc880f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/2510230404 shutdown_connections 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/2510230404 wait complete. 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 Processor -- start 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.322+0000 7fdc8eb0c700 1 -- start start 2024-02-13T03:27:01.326 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8eb0c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8eb0c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8810d2f0 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8c8a8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8c8a8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41336/0 (socket says 172.21.15.80:41336) 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8c8a8700 1 -- 172.21.15.80:0/4099441422 learned_addr learned my addr 172.21.15.80:0/4099441422 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8c8a8700 1 -- 172.21.15.80:0/4099441422 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c00b200 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8c8a8700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdc7c00e040 tx=0x7fdc7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c00da60 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc7c01b8e0 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c01a900 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.323+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8810d4f0 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.324+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8810e100 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.324+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdc7c00f070 con 0x7fdc880703b0 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.325+0000 7fdc857fa700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 0x7fdc6c032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:01.327 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.325+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc7c040f90 con 0x7fdc880703b0 2024-02-13T03:27:01.328 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.325+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc88059100 con 0x7fdc880703b0 2024-02-13T03:27:01.328 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.325+0000 7fdc87fff700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 0x7fdc6c032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:01.328 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.325+0000 7fdc87fff700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 0x7fdc6c032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdc740096a0 tx=0x7fdc74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:01.330 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.329+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc7c016030 con 0x7fdc880703b0 2024-02-13T03:27:01.495 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.494+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdc88102150 con 0x7fdc6c02ffc0 2024-02-13T03:27:01.710 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.709+0000 7fdc857fa700 1 -- 172.21.15.80:0/4099441422 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fdc88102150 con 0x7fdc6c02ffc0 2024-02-13T03:27:01.710 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$JQYQHIhvFvGov3anOnRnk.5OYypOzshpaWaQWezFBGZmGrEC.HUge", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707794821, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 msgr2=0x7fdc6c032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 0x7fdc6c032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fdc740096a0 tx=0x7fdc74006b40).stop 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 msgr2=0x7fdc8810ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdc7c00e040 tx=0x7fdc7c006ee0).stop 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 shutdown_connections 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fdc6c02ffc0 0x7fdc6c032480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:01.713 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 --2- 172.21.15.80:0/4099441422 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdc880703b0 0x7fdc8810ce00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:01.714 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 >> 172.21.15.80:0/4099441422 conn(0x7fdc880f6530 msgr2=0x7fdc880f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:01.714 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 shutdown_connections 2024-02-13T03:27:01.714 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:01.711+0000 7fdc8eb0c700 1 -- 172.21.15.80:0/4099441422 wait complete. 2024-02-13T03:27:02.607 INFO:teuthology.orchestra.run.smithi080.stderr:Fetching dashboard port number... 2024-02-13T03:27:02.607 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp7uo4hw83:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnbfx8aau:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-13T03:27:02.863 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:01.496065+0000 mgr.smithi080.vyzlfg (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:27:02.863 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:01.710709+0000 mon.smithi080 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:03.175 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.173+0000 7f5475e87700 1 Processor -- start 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.173+0000 7f5475e87700 1 -- start start 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f5475e87700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f5475e87700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470072980 con 0x7f5470072560 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41338/0 (socket says 172.21.15.80:41338) 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546f7fe700 1 -- 172.21.15.80:0/4043668004 learned_addr learned my addr 172.21.15.80:0/4043668004 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546f7fe700 1 -- 172.21.15.80:0/4043668004 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470070ff0 con 0x7f5470072560 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546f7fe700 1 --2- 172.21.15.80:0/4043668004 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f545800b5f0 tx=0x7f5458006ee0).ready entity=mon.0 client_cookie=4a99ca04c2d605eb server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:03.176 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546e7fc700 1 -- 172.21.15.80:0/4043668004 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f545800d600 con 0x7f5470072560 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546e7fc700 1 -- 172.21.15.80:0/4043668004 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f545800dc30 con 0x7f5470072560 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.174+0000 7f546e7fc700 1 -- 172.21.15.80:0/4043668004 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5458011d30 con 0x7f5470072560 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 -- 172.21.15.80:0/4043668004 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 msgr2=0x7f5470070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 --2- 172.21.15.80:0/4043668004 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f545800b5f0 tx=0x7f5458006ee0).stop 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 -- 172.21.15.80:0/4043668004 shutdown_connections 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 --2- 172.21.15.80:0/4043668004 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f5470070b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 -- 172.21.15.80:0/4043668004 >> 172.21.15.80:0/4043668004 conn(0x7f54700f76a0 msgr2=0x7f54700f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 -- 172.21.15.80:0/4043668004 shutdown_connections 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.175+0000 7f5475e87700 1 -- 172.21.15.80:0/4043668004 wait complete. 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f5475e87700 1 Processor -- start 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f5475e87700 1 -- start start 2024-02-13T03:27:03.178 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f5475e87700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f5475e87700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547010d3c0 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f546f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f546f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41342/0 (socket says 172.21.15.80:41342) 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f546f7fe700 1 -- 172.21.15.80:0/1380017730 learned_addr learned my addr 172.21.15.80:0/1380017730 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f546f7fe700 1 -- 172.21.15.80:0/1380017730 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545800b200 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.176+0000 7f546f7fe700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5458006c70 tx=0x7f5458006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f545800d920 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547010d5c0 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f545800da80 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f545801a730 con 0x7f5470072560 2024-02-13T03:27:03.179 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547010e1d0 con 0x7f5470072560 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.177+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f545800f070 con 0x7f5470072560 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.178+0000 7f5467fff700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 0x7f54500324e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.178+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5458042510 con 0x7f5470072560 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.178+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5470059100 con 0x7f5470072560 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.178+0000 7f546effd700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 0x7f54500324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:03.180 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.178+0000 7f546effd700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 0x7f54500324e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f54600096a0 tx=0x7f5460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:03.182 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.182+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5458025460 con 0x7f5470072560 2024-02-13T03:27:03.331 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.330+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5470045ad0 con 0x7f5470072560 2024-02-13T03:27:03.331 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.330+0000 7f5467fff700 1 -- 172.21.15.80:0/1380017730 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5458022910 con 0x7f5470072560 2024-02-13T03:27:03.331 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 8443 2024-02-13T03:27:03.333 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 msgr2=0x7f54500324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 0x7f54500324e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f54600096a0 tx=0x7f5460006b40).stop 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 msgr2=0x7f547010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5458006c70 tx=0x7f5458006ee0).stop 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 shutdown_connections 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5450030020 0x7f54500324e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 --2- 172.21.15.80:0/1380017730 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5470072560 0x7f547010ced0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 >> 172.21.15.80:0/1380017730 conn(0x7f54700f76a0 msgr2=0x7f54700f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 shutdown_connections 2024-02-13T03:27:03.334 INFO:teuthology.orchestra.run.smithi080.stderr:/usr/bin/ceph: 2024-02-13T03:27:03.332+0000 7f5475e87700 1 -- 172.21.15.80:0/1380017730 wait complete. 2024-02-13T03:27:03.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:03 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:02.714320+0000 mon.smithi080 (mon.0) 110 : cluster [DBG] mgrmap e17: smithi080.vyzlfg(active, since 6s) 2024-02-13T03:27:03.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:03 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:03.331883+0000 mon.smithi080 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.80:0/1380017730' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-13T03:27:03.981 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-13T03:27:03.990 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: disabled 2024-02-13T03:27:03.990 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: systemctl is-active firewalld.service 2024-02-13T03:27:03.997 INFO:teuthology.orchestra.run.smithi080.stderr:systemctl: inactive 2024-02-13T03:27:03.998 INFO:teuthology.orchestra.run.smithi080.stderr:firewalld.service is not enabled 2024-02-13T03:27:03.998 INFO:teuthology.orchestra.run.smithi080.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-13T03:27:04.000 INFO:teuthology.orchestra.run.smithi080.stderr:Ceph Dashboard is now available at: 2024-02-13T03:27:04.000 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.000 INFO:teuthology.orchestra.run.smithi080.stderr: URL: https://smithi080.front.sepia.ceph.com:8443/ 2024-02-13T03:27:04.001 INFO:teuthology.orchestra.run.smithi080.stderr: User: admin 2024-02-13T03:27:04.001 INFO:teuthology.orchestra.run.smithi080.stderr: Password: ez47e2zlc9 2024-02-13T03:27:04.001 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.001 INFO:teuthology.orchestra.run.smithi080.stderr:You can access the Ceph CLI with: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: ceph telemetry on 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr:For more information see: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:27:04.002 INFO:teuthology.orchestra.run.smithi080.stderr:Bootstrap complete. 2024-02-13T03:27:04.003 INFO:teuthology.orchestra.run.smithi080.stderr:Releasing lock 139809763322344 on /run/cephadm/84391b2a-ca1f-11ee-95b9-87774f69a715.lock 2024-02-13T03:27:04.003 INFO:teuthology.orchestra.run.smithi080.stderr:Lock 139809763322344 released on /run/cephadm/84391b2a-ca1f-11ee-95b9-87774f69a715.lock 2024-02-13T03:27:04.036 INFO:tasks.cephadm:Fetching config... 2024-02-13T03:27:04.036 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:27:04.036 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-13T03:27:04.094 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-13T03:27:04.095 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:27:04.095 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-13T03:27:04.152 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-13T03:27:04.152 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:27:04.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/mon.smithi080/keyring of=/dev/stdout 2024-02-13T03:27:04.219 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-13T03:27:04.219 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-02-13T03:27:04.219 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-13T03:27:04.276 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-13T03:27:04.277 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb/Ii27rT46N558ce5+igpaOoTc18zSE9HL3KV+/ElBQpxVIGI9FRiFRFB2zZ+79SLaA0pvoIxJlzo8ZOk/aTDQIL8bgioMS6Ydou9bEg44PZGsTuuREJGdnSUVmg816N2tM4fZxDPUF+Ae/TmXXGL+XawpwX+MwDWEmuX7v0VmoLUnllAVvhB9Qfq90mEZLytYO6og/PlZRnbQD/saqIyV6UAENM1Gw/KqM+/VSi/MxVrNMXBy2C5x0RoxY0JhvJPmiFyDYxwp3h680GXnwZdeK9sgYhdOdCsQgQMJQclwvSAKbi+lT7qdkN5IGyh8+NXPzQ0wLtElVNeyV5vfZv/cMJ5fDLvSw4Kg5M8MzFMVB5wIMWz+qpeTxneZAxEzyO3YP3wrxSf2YIrsII1y005L2CFyjGGR1LXASsdKFW5JDnpaTDzpIiQXoGLK9yfJBTzyou6E6G2yYoIt2u/KaNaL8zxutpnUjmWLXWFrN3IbacDwIBkAAGLZ7x+8+1xfVM= ceph-84391b2a-ca1f-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T03:27:04.356 INFO:teuthology.orchestra.run.smithi080.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb/Ii27rT46N558ce5+igpaOoTc18zSE9HL3KV+/ElBQpxVIGI9FRiFRFB2zZ+79SLaA0pvoIxJlzo8ZOk/aTDQIL8bgioMS6Ydou9bEg44PZGsTuuREJGdnSUVmg816N2tM4fZxDPUF+Ae/TmXXGL+XawpwX+MwDWEmuX7v0VmoLUnllAVvhB9Qfq90mEZLytYO6og/PlZRnbQD/saqIyV6UAENM1Gw/KqM+/VSi/MxVrNMXBy2C5x0RoxY0JhvJPmiFyDYxwp3h680GXnwZdeK9sgYhdOdCsQgQMJQclwvSAKbi+lT7qdkN5IGyh8+NXPzQ0wLtElVNeyV5vfZv/cMJ5fDLvSw4Kg5M8MzFMVB5wIMWz+qpeTxneZAxEzyO3YP3wrxSf2YIrsII1y005L2CFyjGGR1LXASsdKFW5JDnpaTDzpIiQXoGLK9yfJBTzyou6E6G2yYoIt2u/KaNaL8zxutpnUjmWLXWFrN3IbacDwIBkAAGLZ7x+8+1xfVM= ceph-84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:27:04.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb/Ii27rT46N558ce5+igpaOoTc18zSE9HL3KV+/ElBQpxVIGI9FRiFRFB2zZ+79SLaA0pvoIxJlzo8ZOk/aTDQIL8bgioMS6Ydou9bEg44PZGsTuuREJGdnSUVmg816N2tM4fZxDPUF+Ae/TmXXGL+XawpwX+MwDWEmuX7v0VmoLUnllAVvhB9Qfq90mEZLytYO6og/PlZRnbQD/saqIyV6UAENM1Gw/KqM+/VSi/MxVrNMXBy2C5x0RoxY0JhvJPmiFyDYxwp3h680GXnwZdeK9sgYhdOdCsQgQMJQclwvSAKbi+lT7qdkN5IGyh8+NXPzQ0wLtElVNeyV5vfZv/cMJ5fDLvSw4Kg5M8MzFMVB5wIMWz+qpeTxneZAxEzyO3YP3wrxSf2YIrsII1y005L2CFyjGGR1LXASsdKFW5JDnpaTDzpIiQXoGLK9yfJBTzyou6E6G2yYoIt2u/KaNaL8zxutpnUjmWLXWFrN3IbacDwIBkAAGLZ7x+8+1xfVM= ceph-84391b2a-ca1f-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T03:27:04.411 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb/Ii27rT46N558ce5+igpaOoTc18zSE9HL3KV+/ElBQpxVIGI9FRiFRFB2zZ+79SLaA0pvoIxJlzo8ZOk/aTDQIL8bgioMS6Ydou9bEg44PZGsTuuREJGdnSUVmg816N2tM4fZxDPUF+Ae/TmXXGL+XawpwX+MwDWEmuX7v0VmoLUnllAVvhB9Qfq90mEZLytYO6og/PlZRnbQD/saqIyV6UAENM1Gw/KqM+/VSi/MxVrNMXBy2C5x0RoxY0JhvJPmiFyDYxwp3h680GXnwZdeK9sgYhdOdCsQgQMJQclwvSAKbi+lT7qdkN5IGyh8+NXPzQ0wLtElVNeyV5vfZv/cMJ5fDLvSw4Kg5M8MzFMVB5wIMWz+qpeTxneZAxEzyO3YP3wrxSf2YIrsII1y005L2CFyjGGR1LXASsdKFW5JDnpaTDzpIiQXoGLK9yfJBTzyou6E6G2yYoIt2u/KaNaL8zxutpnUjmWLXWFrN3IbacDwIBkAAGLZ7x+8+1xfVM= ceph-84391b2a-ca1f-11ee-95b9-87774f69a715 2024-02-13T03:27:04.423 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 -- 172.21.15.80:0/3094983106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 msgr2=0x7f2a0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 --2- 172.21.15.80:0/3094983106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c0f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f29fc009fb0 tx=0x7f29fc009b30).stop 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 -- 172.21.15.80:0/3094983106 shutdown_connections 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 --2- 172.21.15.80:0/3094983106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c0f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 -- 172.21.15.80:0/3094983106 >> 172.21.15.80:0/3094983106 conn(0x7f2a0c0f1c70 msgr2=0x7f2a0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 -- 172.21.15.80:0/3094983106 shutdown_connections 2024-02-13T03:27:06.996 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.995+0000 7f2a1116f700 1 -- 172.21.15.80:0/3094983106 wait complete. 2024-02-13T03:27:06.998 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a1116f700 1 Processor -- start 2024-02-13T03:27:06.998 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a1116f700 1 -- start start 2024-02-13T03:27:06.998 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a1116f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:06.998 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a1116f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c069e20 con 0x7f2a0c0f6510 2024-02-13T03:27:06.998 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a0ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a0ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41354/0 (socket says 172.21.15.80:41354) 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a0ad9d700 1 -- 172.21.15.80:0/76732389 learned_addr learned my addr 172.21.15.80:0/76732389 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a0ad9d700 1 -- 172.21.15.80:0/76732389 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc005040 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.996+0000 7f2a0ad9d700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f29fc005010 tx=0x7f29fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc009320 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c069fc0 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29fc013380 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c06a450 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29fc014e50 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.997+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29fc0263a0 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.998+0000 7f29f37fe700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 0x7f29f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.998+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29fc042040 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.998+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c059100 con 0x7f2a0c0f6510 2024-02-13T03:27:06.999 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.998+0000 7f2a0a59c700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 0x7f29f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:07.000 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:06.998+0000 7f2a0a59c700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 0x7f29f40323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2a000096a0 tx=0x7f2a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:07.003 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.002+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29fc012920 con 0x7f2a0c0f6510 2024-02-13T03:27:07.192 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.191+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2a0c045ad0 con 0x7f2a0c0f6510 2024-02-13T03:27:07.195 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.195+0000 7f29f37fe700 1 -- 172.21.15.80:0/76732389 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f29fc02b020 con 0x7f2a0c0f6510 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 msgr2=0x7f29f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 0x7f29f40323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2a000096a0 tx=0x7f2a00006b40).stop 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 msgr2=0x7f2a0c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f29fc005010 tx=0x7f29fc00b040).stop 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 shutdown_connections 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f29f402ff10 0x7f29f40323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 --2- 172.21.15.80:0/76732389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0c0f6510 0x7f2a0c06b9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 >> 172.21.15.80:0/76732389 conn(0x7f2a0c0f1c70 msgr2=0x7f2a0c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 shutdown_connections 2024-02-13T03:27:07.198 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:07.197+0000 7f2a1116f700 1 -- 172.21.15.80:0/76732389 wait complete. 2024-02-13T03:27:07.855 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:06.565305+0000 mon.smithi080 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:07.855 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:06.827948+0000 mon.smithi080 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:07.855 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:07.196277+0000 mon.smithi080 (mon.0) 114 : audit [INF] from='client.? 172.21.15.80:0/76732389' entity='client.admin' 2024-02-13T03:27:07.874 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2024-02-13T03:27:07.874 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:27:07.874 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2024-02-13T03:27:07.896 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-02-13T03:27:07.897 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T03:27:07.957 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2024-02-13T03:27:07.957 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph orch host add smithi112 2024-02-13T03:27:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:27:06.834251+0000 mgr.smithi080.vyzlfg (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi080 on smithi080 2024-02-13T03:27:18.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:16.723791+0000 mgr.smithi080.vyzlfg (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:20.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:19 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:18.724185+0000 mgr.smithi080.vyzlfg (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:22.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:20.724583+0000 mgr.smithi080.vyzlfg (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:24.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:22.724906+0000 mgr.smithi080.vyzlfg (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:26.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:24.725230+0000 mgr.smithi080.vyzlfg (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:28.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:26.725509+0000 mgr.smithi080.vyzlfg (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:30.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:28.725811+0000 mgr.smithi080.vyzlfg (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 -- 172.21.15.112:0/218094616 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 msgr2=0x7f658c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 --2- 172.21.15.112:0/218094616 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c0f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f6588009fb0 tx=0x7f6588009b30).stop 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 -- 172.21.15.112:0/218094616 shutdown_connections 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 --2- 172.21.15.112:0/218094616 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c0f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 -- 172.21.15.112:0/218094616 >> 172.21.15.112:0/218094616 conn(0x7f658c0f09a0 msgr2=0x7f658c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 -- 172.21.15.112:0/218094616 shutdown_connections 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 -- 172.21.15.112:0/218094616 wait complete. 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.926+0000 7f6592433700 1 Processor -- start 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6592433700 1 -- start start 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6592433700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:31.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6592433700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c106ac0 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6591431700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6591431700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42408/0 (socket says 172.21.15.112:42408) 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6591431700 1 -- 172.21.15.112:0/124661679 learned_addr learned my addr 172.21.15.112:0/124661679 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.927+0000 7f6591431700 1 -- 172.21.15.112:0/124661679 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588005040 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6591431700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f65880089f0 tx=0x7f658800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6588009320 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6588013380 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c106cc0 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6588014e20 con 0x7f658c0f51a0 2024-02-13T03:27:31.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.928+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c107150 con 0x7f658c0f51a0 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.929+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c045ad0 con 0x7f658c0f51a0 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.929+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65880095f0 con 0x7f658c0f51a0 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.929+0000 7f6581ffb700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 0x7f65780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.929+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6588040ec0 con 0x7f658c0f51a0 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.929+0000 7f6590c30700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 0x7f65780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:31.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.930+0000 7f6590c30700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 0x7f65780323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f657c0096a0 tx=0x7f657c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:31.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:31.934+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6588014420 con 0x7f658c0f51a0 2024-02-13T03:27:32.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.104+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7f658c1087e0 con 0x7f657802ff10 2024-02-13T03:27:32.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:30.726049+0000 mgr.smithi080.vyzlfg (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:32.698 INFO:teuthology.orchestra.run.smithi112.stdout:Added host 'smithi112' 2024-02-13T03:27:32.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.697+0000 7f6581ffb700 1 -- 172.21.15.112:0/124661679 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f658c1087e0 con 0x7f657802ff10 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 msgr2=0x7f65780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 0x7f65780323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f657c0096a0 tx=0x7f657c006b40).stop 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 msgr2=0x7f658c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f65880089f0 tx=0x7f658800b040).stop 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 shutdown_connections 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f657802ff10 0x7f65780323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 --2- 172.21.15.112:0/124661679 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f658c0f51a0 0x7f658c1065d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:32.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 >> 172.21.15.112:0/124661679 conn(0x7f658c0f09a0 msgr2=0x7f658c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:32.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 shutdown_connections 2024-02-13T03:27:32.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:32.699+0000 7f6592433700 1 -- 172.21.15.112:0/124661679 wait complete. 2024-02-13T03:27:33.349 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-13T03:27:34.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:32.106187+0000 mgr.smithi080.vyzlfg (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:27:34.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:32.697727+0000 mon.smithi080 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:34.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:27:32.698112+0000 mgr.smithi080.vyzlfg (mgr.14162) 15 : cephadm [INF] Added host smithi112 2024-02-13T03:27:35.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 -- 172.21.15.112:0/17730978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 msgr2=0x7f2b7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:35.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/17730978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c0f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2b70009fb0 tx=0x7f2b70009b30).stop 2024-02-13T03:27:35.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 -- 172.21.15.112:0/17730978 shutdown_connections 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/17730978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c0f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 -- 172.21.15.112:0/17730978 >> 172.21.15.112:0/17730978 conn(0x7f2b7c0f1c70 msgr2=0x7f2b7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 -- 172.21.15.112:0/17730978 shutdown_connections 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.118+0000 7f2b82e0e700 1 -- 172.21.15.112:0/17730978 wait complete. 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b82e0e700 1 Processor -- start 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b82e0e700 1 -- start start 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b82e0e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b82e0e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c069e20 con 0x7f2b7c0f6510 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b80baa700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b80baa700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42440/0 (socket says 172.21.15.112:42440) 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.119+0000 7f2b80baa700 1 -- 172.21.15.112:0/1356599689 learned_addr learned my addr 172.21.15.112:0/1356599689 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b80baa700 1 -- 172.21.15.112:0/1356599689 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70005040 con 0x7f2b7c0f6510 2024-02-13T03:27:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b80baa700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2b70005010 tx=0x7f2b7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b70009320 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c069fc0 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b70013380 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b70014e50 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.120+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c06a450 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b700263a0 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c059100 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b797fa700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 0x7f2b6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b70041220 con 0x7f2b7c0f6510 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b7bfff700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 0x7f2b6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.121+0000 7f2b7bfff700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 0x7f2b6c0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2b680096a0 tx=0x7f2b68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:35.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.125+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b7001ccf0 con 0x7f2b7c0f6510 2024-02-13T03:27:35.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:32.726308+0000 mgr.smithi080.vyzlfg (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:35.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.294+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b7c06aca0 con 0x7f2b6c02ff10 2024-02-13T03:27:35.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.296+0000 7f2b797fa700 1 -- 172.21.15.112:0/1356599689 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f2b7c06aca0 con 0x7f2b6c02ff10 2024-02-13T03:27:35.297 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:27:35.297 INFO:teuthology.orchestra.run.smithi112.stdout:[{"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}, {"addr": "smithi112", "hostname": "smithi112", "labels": [], "status": ""}] 2024-02-13T03:27:35.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 msgr2=0x7f2b6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:35.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 0x7f2b6c0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2b680096a0 tx=0x7f2b68006b40).stop 2024-02-13T03:27:35.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 msgr2=0x7f2b7c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:35.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2b70005010 tx=0x7f2b7000b040).stop 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 shutdown_connections 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b6c02ff10 0x7f2b6c0323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 --2- 172.21.15.112:0/1356599689 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b7c0f6510 0x7f2b7c06b9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 >> 172.21.15.112:0/1356599689 conn(0x7f2b7c0f1c70 msgr2=0x7f2b7c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 shutdown_connections 2024-02-13T03:27:35.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:35.297+0000 7f2b82e0e700 1 -- 172.21.15.112:0/1356599689 wait complete. 2024-02-13T03:27:35.940 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-13T03:27:35.940 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-13T03:27:37.043 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:34.726554+0000 mgr.smithi080.vyzlfg (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:37.043 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:35.296634+0000 mgr.smithi080.vyzlfg (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T03:27:37.788 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 -- 172.21.15.80:0/3340246916 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 msgr2=0x7efe280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:37.788 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 --2- 172.21.15.80:0/3340246916 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe280f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efe18009fb0 tx=0x7efe18009b30).stop 2024-02-13T03:27:37.788 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 -- 172.21.15.80:0/3340246916 shutdown_connections 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 --2- 172.21.15.80:0/3340246916 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe280f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 -- 172.21.15.80:0/3340246916 >> 172.21.15.80:0/3340246916 conn(0x7efe280f0b70 msgr2=0x7efe280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 -- 172.21.15.80:0/3340246916 shutdown_connections 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 -- 172.21.15.80:0/3340246916 wait complete. 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.782+0000 7efe30319700 1 Processor -- start 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe30319700 1 -- start start 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe30319700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe30319700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28060220 con 0x7efe280f5410 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe2e0b5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe2e0b5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:41252/0 (socket says 172.21.15.80:41252) 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe2e0b5700 1 -- 172.21.15.80:0/4098107740 learned_addr learned my addr 172.21.15.80:0/4098107740 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-02-13T03:27:37.789 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe2e0b5700 1 -- 172.21.15.80:0/4098107740 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18005040 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe2e0b5700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efe18005010 tx=0x7efe1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe18009320 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe18013380 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe28060420 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe18014e50 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.783+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe28109bb0 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.784+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe180263a0 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.784+0000 7efe1effd700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 0x7efe140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.784+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe28059100 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.784+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe18040e50 con 0x7efe280f5410 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.784+0000 7efe2d8b4700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 0x7efe140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:37.790 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.785+0000 7efe2d8b4700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 0x7efe140323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efe240096a0 tx=0x7efe24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:37.791 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.788+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe18012920 con 0x7efe280f5410 2024-02-13T03:27:37.942 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:37.943+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efe280fafa0 con 0x7efe280f5410 2024-02-13T03:27:38.707 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.706+0000 7efe1effd700 1 -- 172.21.15.80:0/4098107740 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7efe180126f0 con 0x7efe280f5410 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 msgr2=0x7efe140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 0x7efe140323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efe240096a0 tx=0x7efe24006b40).stop 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 msgr2=0x7efe28109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efe18005010 tx=0x7efe1800b040).stop 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 shutdown_connections 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efe1402ff10 0x7efe140323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 --2- 172.21.15.80:0/4098107740 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efe280f5410 0x7efe28109500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 >> 172.21.15.80:0/4098107740 conn(0x7efe280f0b70 msgr2=0x7efe280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 shutdown_connections 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:2024-02-13T03:27:38.709+0000 7efe30319700 1 -- 172.21.15.80:0/4098107740 wait complete. 2024-02-13T03:27:38.710 INFO:teuthology.orchestra.run.smithi080.stderr:adjusted tunables profile to default 2024-02-13T03:27:39.116 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:36.726835+0000 mgr.smithi080.vyzlfg (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:39.116 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:37.943730+0000 mon.smithi080 (mon.0) 116 : audit [INF] from='client.? 172.21.15.80:0/4098107740' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-13T03:27:39.403 INFO:tasks.cephadm:Adding mon.smithi080 on smithi080 2024-02-13T03:27:39.403 INFO:tasks.cephadm:Adding mon.smithi112 on smithi112 2024-02-13T03:27:39.403 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph orch apply mon '2;smithi080:172.21.15.80=smithi080;smithi112:172.21.15.112=smithi112' 2024-02-13T03:27:40.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:38.707418+0000 mon.smithi080 (mon.0) 117 : audit [INF] from='client.? 172.21.15.80:0/4098107740' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-13T03:27:40.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:38.707469+0000 mon.smithi080 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-13T03:27:41.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:38.727170+0000 mgr.smithi080.vyzlfg (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:41.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3515819563 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 msgr2=0x7f4fb80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3515819563 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb80f66c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4fac009fb0 tx=0x7f4fac009b30).stop 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3515819563 shutdown_connections 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3515819563 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb80f66c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3515819563 >> 172.21.15.112:0/3515819563 conn(0x7f4fb80f1aa0 msgr2=0x7f4fb80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3515819563 shutdown_connections 2024-02-13T03:27:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3515819563 wait complete. 2024-02-13T03:27:41.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 Processor -- start 2024-02-13T03:27:41.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.218+0000 7f4fbdd8e700 1 -- start start 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbdd8e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbdd8e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb8106ac0 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbcd8c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbcd8c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42456/0 (socket says 172.21.15.112:42456) 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbcd8c700 1 -- 172.21.15.112:0/3035253298 learned_addr learned my addr 172.21.15.112:0/3035253298 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbcd8c700 1 -- 172.21.15.112:0/3035253298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac005040 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.219+0000 7f4fbcd8c700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4fac0089f0 tx=0x7f4fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fac009320 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb8106cc0 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fac013380 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fac014e20 con 0x7f4fb80f62a0 2024-02-13T03:27:41.221 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb8107150 con 0x7f4fb80f62a0 2024-02-13T03:27:41.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fac014420 con 0x7f4fb80f62a0 2024-02-13T03:27:41.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.220+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb8045ad0 con 0x7f4fb80f62a0 2024-02-13T03:27:41.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.221+0000 7f4fb57fa700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 0x7f4fa80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:41.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.221+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fac040ed0 con 0x7f4fb80f62a0 2024-02-13T03:27:41.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.221+0000 7f4fb7fff700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 0x7f4fa80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:41.223 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.221+0000 7f4fb7fff700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 0x7f4fa80323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4fa40096a0 tx=0x7f4fa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:41.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.225+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fac014790 con 0x7f4fb80f62a0 2024-02-13T03:27:41.399 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.399+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 --> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi080:172.21.15.80=smithi080;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fb80f2c50 con 0x7f4fa802ff10 2024-02-13T03:27:41.404 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.404+0000 7f4fb57fa700 1 -- 172.21.15.112:0/3035253298 <== mgr.14162 v2:172.21.15.80:6800/2233275020 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4fb80f2c50 con 0x7f4fa802ff10 2024-02-13T03:27:41.405 INFO:teuthology.orchestra.run.smithi112.stdout:Scheduled mon update... 2024-02-13T03:27:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 msgr2=0x7f4fa80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 0x7f4fa80323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4fa40096a0 tx=0x7f4fa4006b40).stop 2024-02-13T03:27:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 msgr2=0x7f4fb81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4fac0089f0 tx=0x7f4fac00b040).stop 2024-02-13T03:27:41.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 shutdown_connections 2024-02-13T03:27:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4fa802ff10 0x7f4fa80323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 --2- 172.21.15.112:0/3035253298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fb80f62a0 0x7f4fb81065d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 >> 172.21.15.112:0/3035253298 conn(0x7f4fb80f1aa0 msgr2=0x7f4fb80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 shutdown_connections 2024-02-13T03:27:41.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:41.405+0000 7f4fbdd8e700 1 -- 172.21.15.112:0/3035253298 wait complete. 2024-02-13T03:27:42.028 DEBUG:teuthology.orchestra.run.smithi112:mon.smithi112> sudo journalctl -f -n 0 -u ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112.service 2024-02-13T03:27:42.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:27:42.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:27:42.060 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:-- Logs begin at Tue 2024-02-13 03:15:22 UTC. -- 2024-02-13T03:27:42.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:40.727469+0000 mgr.smithi080.vyzlfg (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:42.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:41.401128+0000 mgr.smithi080.vyzlfg (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi080:172.21.15.80=smithi080;smithi112:172.21.15.112=smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T03:27:42.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:27:41.402476+0000 mgr.smithi080.vyzlfg (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi080:172.21.15.80=smithi080;smithi112:172.21.15.112=smithi112;count:2 2024-02-13T03:27:42.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:41.404955+0000 mon.smithi080 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:27:43.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- 172.21.15.112:0/2455115737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 msgr2=0x7f0c080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 --2- 172.21.15.112:0/2455115737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c080f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0bfc009fb0 tx=0x7f0bfc009b30).stop 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- 172.21.15.112:0/2455115737 shutdown_connections 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 --2- 172.21.15.112:0/2455115737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c080f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- 172.21.15.112:0/2455115737 >> 172.21.15.112:0/2455115737 conn(0x7f0c080f0b70 msgr2=0x7f0c080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- 172.21.15.112:0/2455115737 shutdown_connections 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- 172.21.15.112:0/2455115737 wait complete. 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 Processor -- start 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.798+0000 7f0c0f220700 1 -- start start 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0f220700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0f220700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08105bd0 con 0x7f0c080f5410 2024-02-13T03:27:43.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0cfbc700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0cfbc700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47684/0 (socket says 172.21.15.112:47684) 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0cfbc700 1 -- 172.21.15.112:0/3571474340 learned_addr learned my addr 172.21.15.112:0/3571474340 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0cfbc700 1 -- 172.21.15.112:0/3571474340 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc005040 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0cfbc700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0bfc005010 tx=0x7f0bfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bfc009320 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c08105dd0 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bfc013380 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bfc014e50 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.799+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c08106260 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.800+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bfc014420 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.800+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c08045ad0 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.800+0000 7f0c057fa700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 0x7f0bf80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.800+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bfc040f40 con 0x7f0c080f5410 2024-02-13T03:27:43.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.800+0000 7f0c07fff700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 0x7f0bf80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:43.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.801+0000 7f0c07fff700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 0x7f0bf80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0bf40096a0 tx=0x7f0bf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:43.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:43.805+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bfc0095f0 con 0x7f0c080f5410 2024-02-13T03:27:44.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.010+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c08107110 con 0x7f0c080f5410 2024-02-13T03:27:44.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.011+0000 7f0c057fa700 1 -- 172.21.15.112:0/3571474340 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bfc01db70 con 0x7f0c080f5410 2024-02-13T03:27:44.012 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:27:44.012 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:27:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.012+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 msgr2=0x7f0bf80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.012+0000 7f0c0f220700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 0x7f0bf80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0bf40096a0 tx=0x7f0bf4006b40).stop 2024-02-13T03:27:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.012+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 msgr2=0x7f0c081056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:44.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.012+0000 7f0c0f220700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0bfc005010 tx=0x7f0bfc00b040).stop 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 shutdown_connections 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0bf802ff10 0x7f0bf80323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 --2- 172.21.15.112:0/3571474340 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0c080f5410 0x7f0c081056e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 >> 172.21.15.112:0/3571474340 conn(0x7f0c080f0b70 msgr2=0x7f0c080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 shutdown_connections 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:44.013+0000 7f0c0f220700 1 -- 172.21.15.112:0/3571474340 wait complete. 2024-02-13T03:27:44.014 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:27:44.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:42.727849+0000 mgr.smithi080.vyzlfg (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:44.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:44.012478+0000 mon.smithi080 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.112:0/3571474340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:27:45.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:27:45.679 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:27:46.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:44.728187+0000 mgr.smithi080.vyzlfg (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:47.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/2753727203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 msgr2=0x7f3a840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:47.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/2753727203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a840f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f3a78009fb0 tx=0x7f3a78009b30).stop 2024-02-13T03:27:47.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/2753727203 shutdown_connections 2024-02-13T03:27:47.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/2753727203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a840f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:47.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/2753727203 >> 172.21.15.112:0/2753727203 conn(0x7f3a840f0b70 msgr2=0x7f3a840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:47.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/2753727203 shutdown_connections 2024-02-13T03:27:47.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.465+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/2753727203 wait complete. 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a8ae6c700 1 Processor -- start 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a8ae6c700 1 -- start start 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a8ae6c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a8ae6c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84105bd0 con 0x7f3a840f5410 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a88c08700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.466+0000 7f3a88c08700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47706/0 (socket says 172.21.15.112:47706) 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a88c08700 1 -- 172.21.15.112:0/3039119057 learned_addr learned my addr 172.21.15.112:0/3039119057 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a88c08700 1 -- 172.21.15.112:0/3039119057 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a78005040 con 0x7f3a840f5410 2024-02-13T03:27:47.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a88c08700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3a78005010 tx=0x7f3a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a78009320 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a84105dd0 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a78013380 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a78014e50 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.467+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a84106260 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.468+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a78014420 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.468+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a84045ad0 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.468+0000 7f3a817fa700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 0x7f3a740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.468+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a78040f70 con 0x7f3a840f5410 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.468+0000 7f3a83fff700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 0x7f3a740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:47.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.469+0000 7f3a83fff700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 0x7f3a740323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3a700096a0 tx=0x7f3a70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:47.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.473+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a780095f0 con 0x7f3a840f5410 2024-02-13T03:27:47.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.677+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a841070b0 con 0x7f3a840f5410 2024-02-13T03:27:47.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.678+0000 7f3a817fa700 1 -- 172.21.15.112:0/3039119057 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a7801db70 con 0x7f3a840f5410 2024-02-13T03:27:47.679 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:27:47.679 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:27:47.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 msgr2=0x7f3a740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:47.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 0x7f3a740323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3a700096a0 tx=0x7f3a70006b40).stop 2024-02-13T03:27:47.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 msgr2=0x7f3a841056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:47.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3a78005010 tx=0x7f3a7800b040).stop 2024-02-13T03:27:47.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 shutdown_connections 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3a7402ff10 0x7f3a740323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.679+0000 7f3a8ae6c700 1 --2- 172.21.15.112:0/3039119057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3a840f5410 0x7f3a841056e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.680+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 >> 172.21.15.112:0/3039119057 conn(0x7f3a840f0b70 msgr2=0x7f3a840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.680+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 shutdown_connections 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:47.680+0000 7f3a8ae6c700 1 -- 172.21.15.112:0/3039119057 wait complete. 2024-02-13T03:27:47.681 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:27:48.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:46.728440+0000 mgr.smithi080.vyzlfg (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:48.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:47.678987+0000 mon.smithi080 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.112:0/3039119057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:27:49.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:27:49.365 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:27:50.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:48.728812+0000 mgr.smithi080.vyzlfg (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.908+0000 7fa255cbd700 1 -- 172.21.15.112:0/4124387947 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 msgr2=0x7fa2500f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.908+0000 7fa255cbd700 1 --2- 172.21.15.112:0/4124387947 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa2500f66c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa244009fb0 tx=0x7fa244009b30).stop 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.908+0000 7fa255cbd700 1 -- 172.21.15.112:0/4124387947 shutdown_connections 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.908+0000 7fa255cbd700 1 --2- 172.21.15.112:0/4124387947 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa2500f66c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.908+0000 7fa255cbd700 1 -- 172.21.15.112:0/4124387947 >> 172.21.15.112:0/4124387947 conn(0x7fa2500f1aa0 msgr2=0x7fa2500f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 -- 172.21.15.112:0/4124387947 shutdown_connections 2024-02-13T03:27:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 -- 172.21.15.112:0/4124387947 wait complete. 2024-02-13T03:27:51.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 Processor -- start 2024-02-13T03:27:51.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 -- start start 2024-02-13T03:27:51.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:51.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.909+0000 7fa255cbd700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250069e20 con 0x7fa2500f62a0 2024-02-13T03:27:51.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa254cbb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:51.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa254cbb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:59990/0 (socket says 172.21.15.112:59990) 2024-02-13T03:27:51.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa254cbb700 1 -- 172.21.15.112:0/560513368 learned_addr learned my addr 172.21.15.112:0/560513368 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:51.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa254cbb700 1 -- 172.21.15.112:0/560513368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa244005040 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa254cbb700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa2440089f0 tx=0x7fa24400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa244009320 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa250069fc0 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa244013380 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa244014e20 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.910+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25006a450 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.911+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2440263a0 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.911+0000 7fa24d7fa700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 0x7fa2400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.911+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250059100 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.911+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2440411d0 con 0x7fa2500f62a0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.911+0000 7fa24ffff700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 0x7fa2400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:51.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.912+0000 7fa24ffff700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 0x7fa2400323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa23c0096a0 tx=0x7fa23c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:51.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:51.916+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa244012920 con 0x7fa2500f62a0 2024-02-13T03:27:52.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.121+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2500fbe30 con 0x7fa2500f62a0 2024-02-13T03:27:52.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.121+0000 7fa24d7fa700 1 -- 172.21.15.112:0/560513368 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa24402b030 con 0x7fa2500f62a0 2024-02-13T03:27:52.122 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:27:52.122 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 msgr2=0x7fa2400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 0x7fa2400323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa23c0096a0 tx=0x7fa23c006b40).stop 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 msgr2=0x7fa25006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa2440089f0 tx=0x7fa24400b040).stop 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 shutdown_connections 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa24002ff10 0x7fa2400323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 --2- 172.21.15.112:0/560513368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2500f62a0 0x7fa25006b9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 >> 172.21.15.112:0/560513368 conn(0x7fa2500f1aa0 msgr2=0x7fa2500f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:52.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 shutdown_connections 2024-02-13T03:27:52.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:52.123+0000 7fa255cbd700 1 -- 172.21.15.112:0/560513368 wait complete. 2024-02-13T03:27:52.125 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:27:52.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:50.729113+0000 mgr.smithi080.vyzlfg (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:52.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:52.122776+0000 mon.smithi080 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.112:0/560513368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:27:53.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:27:53.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:27:54.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:52.729440+0000 mgr.smithi080.vyzlfg (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 -- 172.21.15.112:0/1938602516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 msgr2=0x7f92940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 --2- 172.21.15.112:0/1938602516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f92940f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9288009fb0 tx=0x7f9288009b30).stop 2024-02-13T03:27:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 -- 172.21.15.112:0/1938602516 shutdown_connections 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 --2- 172.21.15.112:0/1938602516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f92940f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 -- 172.21.15.112:0/1938602516 >> 172.21.15.112:0/1938602516 conn(0x7f92940f0b70 msgr2=0x7f92940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 -- 172.21.15.112:0/1938602516 shutdown_connections 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.497+0000 7f9298e90700 1 -- 172.21.15.112:0/1938602516 wait complete. 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f9298e90700 1 Processor -- start 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f9298e90700 1 -- start start 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f9298e90700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f9298e90700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92941072f0 con 0x7f92940f5410 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f929259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f929259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:59996/0 (socket says 172.21.15.112:59996) 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f929259c700 1 -- 172.21.15.112:0/3051670480 learned_addr learned my addr 172.21.15.112:0/3051670480 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f929259c700 1 -- 172.21.15.112:0/3051670480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288005040 con 0x7f92940f5410 2024-02-13T03:27:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f929259c700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9288005010 tx=0x7f928800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.498+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9288009320 con 0x7f92940f5410 2024-02-13T03:27:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9294060220 con 0x7f92940f5410 2024-02-13T03:27:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9288013380 con 0x7f92940f5410 2024-02-13T03:27:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9288014e50 con 0x7f92940f5410 2024-02-13T03:27:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929410bdf0 con 0x7f92940f5410 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92880263a0 con 0x7f92940f5410 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9282ffd700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 0x7f927c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9294045ad0 con 0x7f92940f5410 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.499+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9288041180 con 0x7f92940f5410 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.500+0000 7f9291d9b700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 0x7f927c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:27:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.500+0000 7f9291d9b700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 0x7f927c0323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f92840096a0 tx=0x7f9284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:27:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.504+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9288012920 con 0x7f92940f5410 2024-02-13T03:27:55.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.709+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f929410c5b0 con 0x7f92940f5410 2024-02-13T03:27:55.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.710+0000 7f9282ffd700 1 -- 172.21.15.112:0/3051670480 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9288012330 con 0x7f92940f5410 2024-02-13T03:27:55.711 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:27:55.711 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:27:55.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 msgr2=0x7f927c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:55.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 0x7f927c0323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f92840096a0 tx=0x7f9284006b40).stop 2024-02-13T03:27:55.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 msgr2=0x7f929410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:27:55.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9288005010 tx=0x7f928800b040).stop 2024-02-13T03:27:55.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 shutdown_connections 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f927c02ff10 0x7f927c0323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 --2- 172.21.15.112:0/3051670480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f92940f5410 0x7f929410b900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 >> 172.21.15.112:0/3051670480 conn(0x7f92940f0b70 msgr2=0x7f92940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 shutdown_connections 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:27:55.711+0000 7f9298e90700 1 -- 172.21.15.112:0/3051670480 wait complete. 2024-02-13T03:27:55.713 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:27:56.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:54.729824+0000 mgr.smithi080.vyzlfg (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:27:56.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:55.710937+0000 mon.smithi080 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.112:0/3051670480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:27:57.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:27:57.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:27:57.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:56.853430+0000 mon.smithi080 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:27:57.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:27:56.861958+0000 mon.smithi080 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:27:58.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:27:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:56.730153+0000 mgr.smithi080.vyzlfg (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:00.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:27:58.730519+0000 mgr.smithi080.vyzlfg (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:02.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:00.730780+0000 mgr.smithi080.vyzlfg (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:04.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:02.730996+0000 mgr.smithi080.vyzlfg (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.286+0000 7f50a20d4700 1 -- 172.21.15.112:0/2594474820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 msgr2=0x7f509c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.286+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2594474820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f508c009fb0 tx=0x7f508c009b30).stop 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.286+0000 7f50a20d4700 1 -- 172.21.15.112:0/2594474820 shutdown_connections 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.286+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2594474820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.286+0000 7f50a20d4700 1 -- 172.21.15.112:0/2594474820 >> 172.21.15.112:0/2594474820 conn(0x7f509c0f0b70 msgr2=0x7f509c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.287+0000 7f50a20d4700 1 -- 172.21.15.112:0/2594474820 shutdown_connections 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.287+0000 7f50a20d4700 1 -- 172.21.15.112:0/2594474820 wait complete. 2024-02-13T03:28:06.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.287+0000 7f50a20d4700 1 Processor -- start 2024-02-13T03:28:06.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.287+0000 7f50a20d4700 1 -- start start 2024-02-13T03:28:06.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f50a20d4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:06.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f50a20d4700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c060240 con 0x7f509c0f5410 2024-02-13T03:28:06.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f509b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:06.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f509b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:38468/0 (socket says 172.21.15.112:38468) 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f509b7fe700 1 -- 172.21.15.112:0/2107416537 learned_addr learned my addr 172.21.15.112:0/2107416537 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f509b7fe700 1 -- 172.21.15.112:0/2107416537 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c005040 con 0x7f509c0f5410 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.288+0000 7f509b7fe700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f508c005010 tx=0x7f508c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.289+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f508c009320 con 0x7f509c0f5410 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.289+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f508c013380 con 0x7f509c0f5410 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.289+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c060440 con 0x7f509c0f5410 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.289+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f508c014e50 con 0x7f509c0f5410 2024-02-13T03:28:06.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.289+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c109bc0 con 0x7f509c0f5410 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f508c014420 con 0x7f509c0f5410 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f5083fff700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 0x7f50840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c045ad0 con 0x7f509c0f5410 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f508c040f40 con 0x7f509c0f5410 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f509affd700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 0x7f50840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:06.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.290+0000 7f509affd700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 0x7f50840323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f50900096a0 tx=0x7f5090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:06.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.295+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f508c01db70 con 0x7f509c0f5410 2024-02-13T03:28:06.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.500+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f509c10a3e0 con 0x7f509c0f5410 2024-02-13T03:28:06.500 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.501+0000 7f5083fff700 1 -- 172.21.15.112:0/2107416537 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f508c01db70 con 0x7f509c0f5410 2024-02-13T03:28:06.501 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:06.501 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:06.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 msgr2=0x7f50840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:06.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 0x7f50840323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f50900096a0 tx=0x7f5090006b40).stop 2024-02-13T03:28:06.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 msgr2=0x7f509c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f508c005010 tx=0x7f508c00b040).stop 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 shutdown_connections 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f508402ff10 0x7f50840323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 --2- 172.21.15.112:0/2107416537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f509c0f5410 0x7f509c109510 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.502+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 >> 172.21.15.112:0/2107416537 conn(0x7f509c0f0b70 msgr2=0x7f509c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.503+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 shutdown_connections 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:06.503+0000 7f50a20d4700 1 -- 172.21.15.112:0/2107416537 wait complete. 2024-02-13T03:28:06.503 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:06.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:04.731221+0000 mgr.smithi080.vyzlfg (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:07.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:06.501090+0000 mon.smithi080 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.112:0/2107416537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:08.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:08.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:08.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:06.731555+0000 mgr.smithi080.vyzlfg (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:09.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 -- 172.21.15.112:0/2608638867 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 msgr2=0x7f6d9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:09.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 --2- 172.21.15.112:0/2608638867 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c0f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6d84009fb0 tx=0x7f6d84009b30).stop 2024-02-13T03:28:09.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 -- 172.21.15.112:0/2608638867 shutdown_connections 2024-02-13T03:28:09.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 --2- 172.21.15.112:0/2608638867 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c0f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:09.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 -- 172.21.15.112:0/2608638867 >> 172.21.15.112:0/2608638867 conn(0x7f6d9c0f1c70 msgr2=0x7f6d9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:09.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 -- 172.21.15.112:0/2608638867 shutdown_connections 2024-02-13T03:28:09.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.928+0000 7f6da0a88700 1 -- 172.21.15.112:0/2608638867 wait complete. 2024-02-13T03:28:09.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6da0a88700 1 Processor -- start 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6da0a88700 1 -- start start 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6da0a88700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6da0a88700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c069e20 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6d9a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6d9a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:38480/0 (socket says 172.21.15.112:38480) 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6d9a59c700 1 -- 172.21.15.112:0/549630398 learned_addr learned my addr 172.21.15.112:0/549630398 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.929+0000 7f6d9a59c700 1 -- 172.21.15.112:0/549630398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84005040 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6d9a59c700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6d84005010 tx=0x7f6d8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d84009320 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c069fc0 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d84013380 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d84014e50 con 0x7f6d9c0f6510 2024-02-13T03:28:09.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.930+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c06a450 con 0x7f6d9c0f6510 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c059100 con 0x7f6d9c0f6510 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d84014420 con 0x7f6d9c0f6510 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6d92ffd700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d84040e60 con 0x7f6d9c0f6510 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6d99d9b700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:09.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.931+0000 7f6d99d9b700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6d8c0096a0 tx=0x7f6d8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:09.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:09.936+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d840095f0 con 0x7f6d9c0f6510 2024-02-13T03:28:10.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.138+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d9c045ad0 con 0x7f6d9c0f6510 2024-02-13T03:28:10.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.138+0000 7f6d92ffd700 1 -- 172.21.15.112:0/549630398 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d8401db70 con 0x7f6d9c0f6510 2024-02-13T03:28:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:10.139 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 msgr2=0x7f6d880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6d8c0096a0 tx=0x7f6d8c006b40).stop 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 msgr2=0x7f6d9c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6d84005010 tx=0x7f6d8400b040).stop 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 shutdown_connections 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 --2- 172.21.15.112:0/549630398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d9c0f6510 0x7f6d9c06b9a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 >> 172.21.15.112:0/549630398 conn(0x7f6d9c0f1c70 msgr2=0x7f6d9c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 shutdown_connections 2024-02-13T03:28:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:10.140+0000 7f6da0a88700 1 -- 172.21.15.112:0/549630398 wait complete. 2024-02-13T03:28:10.142 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:10.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:08.731827+0000 mgr.smithi080.vyzlfg (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:10.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:10.139924+0000 mon.smithi080 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.112:0/549630398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:11.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:11.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:12.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:10.732072+0000 mgr.smithi080.vyzlfg (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 -- 172.21.15.112:0/3046412499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 msgr2=0x7ff9200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3046412499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9200f90f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff910009960 tx=0x7ff910004d10).stop 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 -- 172.21.15.112:0/3046412499 shutdown_connections 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3046412499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9200f90f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 -- 172.21.15.112:0/3046412499 >> 172.21.15.112:0/3046412499 conn(0x7ff9200f0b70 msgr2=0x7ff9200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 -- 172.21.15.112:0/3046412499 shutdown_connections 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.494+0000 7ff927e1a700 1 -- 172.21.15.112:0/3046412499 wait complete. 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff927e1a700 1 Processor -- start 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff927e1a700 1 -- start start 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff927e1a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff927e1a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff920106ce0 con 0x7ff920060280 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff925bb6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff925bb6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:59624/0 (socket says 172.21.15.112:59624) 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.495+0000 7ff925bb6700 1 -- 172.21.15.112:0/3342846246 learned_addr learned my addr 172.21.15.112:0/3342846246 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff925bb6700 1 -- 172.21.15.112:0/3342846246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff910009640 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff925bb6700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff910007b20 tx=0x7ff910004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff910018070 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff920106ee0 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff910007e60 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff91001d420 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.496+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff920107370 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff910013630 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920045ad0 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff91e7fc700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 0x7ff908032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff910041590 con 0x7ff920060280 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff9253b5700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 0x7ff908032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.497+0000 7ff9253b5700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 0x7ff908032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7ff9140096a0 tx=0x7ff914006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.501+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff910004810 con 0x7ff920060280 2024-02-13T03:28:13.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.706+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff920108220 con 0x7ff920060280 2024-02-13T03:28:13.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.707+0000 7ff91e7fc700 1 -- 172.21.15.112:0/3342846246 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff91001a2d0 con 0x7ff920060280 2024-02-13T03:28:13.708 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:13.708 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.708+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 msgr2=0x7ff908032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 0x7ff908032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7ff9140096a0 tx=0x7ff914006b40).stop 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 msgr2=0x7ff9201067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff910007b20 tx=0x7ff910004d10).stop 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 shutdown_connections 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff90802ff60 0x7ff908032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 --2- 172.21.15.112:0/3342846246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff920060280 0x7ff9201067f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 >> 172.21.15.112:0/3342846246 conn(0x7ff9200f0b70 msgr2=0x7ff9200f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 shutdown_connections 2024-02-13T03:28:13.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:13.709+0000 7ff927e1a700 1 -- 172.21.15.112:0/3342846246 wait complete. 2024-02-13T03:28:13.711 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:14.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:12.732400+0000 mgr.smithi080.vyzlfg (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:14.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:13.708332+0000 mon.smithi080 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.112:0/3342846246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:15.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:15.410 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:16.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:14.732800+0000 mgr.smithi080.vyzlfg (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.281+0000 7f2d68365700 1 -- 172.21.15.112:0/3352983011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 msgr2=0x7f2d600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.281+0000 7f2d68365700 1 --2- 172.21.15.112:0/3352983011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2d50009fb0 tx=0x7f2d50009b30).stop 2024-02-13T03:28:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- 172.21.15.112:0/3352983011 shutdown_connections 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 --2- 172.21.15.112:0/3352983011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- 172.21.15.112:0/3352983011 >> 172.21.15.112:0/3352983011 conn(0x7f2d600f0b70 msgr2=0x7f2d600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- 172.21.15.112:0/3352983011 shutdown_connections 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- 172.21.15.112:0/3352983011 wait complete. 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 Processor -- start 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- start start 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d68365700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600fb960 con 0x7f2d600f5410 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.282+0000 7f2d66101700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d66101700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:59650/0 (socket says 172.21.15.112:59650) 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d66101700 1 -- 172.21.15.112:0/2210768053 learned_addr learned my addr 172.21.15.112:0/2210768053 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d66101700 1 -- 172.21.15.112:0/2210768053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d50005040 con 0x7f2d600f5410 2024-02-13T03:28:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d66101700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2d50005010 tx=0x7f2d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d50009320 con 0x7f2d600f5410 2024-02-13T03:28:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d50013380 con 0x7f2d600f5410 2024-02-13T03:28:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d600fbb00 con 0x7f2d600f5410 2024-02-13T03:28:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d50014e50 con 0x7f2d600f5410 2024-02-13T03:28:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.283+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d600fbf90 con 0x7f2d600f5410 2024-02-13T03:28:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.284+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d50014420 con 0x7f2d600f5410 2024-02-13T03:28:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.284+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d60059100 con 0x7f2d600f5410 2024-02-13T03:28:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.284+0000 7f2d56ffd700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 0x7f2d4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.284+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d50040eb0 con 0x7f2d600f5410 2024-02-13T03:28:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.284+0000 7f2d65900700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 0x7f2d4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.285+0000 7f2d65900700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 0x7f2d4c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2d5c0096a0 tx=0x7f2d5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.289+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d500095f0 con 0x7f2d600f5410 2024-02-13T03:28:17.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.494+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d600fafa0 con 0x7f2d600f5410 2024-02-13T03:28:17.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.494+0000 7f2d56ffd700 1 -- 172.21.15.112:0/2210768053 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d5001db70 con 0x7f2d600f5410 2024-02-13T03:28:17.495 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:17.495 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 msgr2=0x7f2d4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 0x7f2d4c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2d5c0096a0 tx=0x7f2d5c006b40).stop 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 msgr2=0x7f2d600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2d50005010 tx=0x7f2d5000b040).stop 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 shutdown_connections 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d4c02ff10 0x7f2d4c0323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 --2- 172.21.15.112:0/2210768053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d600f5410 0x7f2d600fd4e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 >> 172.21.15.112:0/2210768053 conn(0x7f2d600f0b70 msgr2=0x7f2d6005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 shutdown_connections 2024-02-13T03:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:17.496+0000 7f2d68365700 1 -- 172.21.15.112:0/2210768053 wait complete. 2024-02-13T03:28:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:18.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:16.733214+0000 mgr.smithi080.vyzlfg (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:18.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:17.495625+0000 mon.smithi080 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.112:0/2210768053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:19.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:19.119 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:20.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:18.733500+0000 mgr.smithi080.vyzlfg (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:21.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- 172.21.15.112:0/118477265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 msgr2=0x7f57c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:21.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 --2- 172.21.15.112:0/118477265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c00f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f57b0009fb0 tx=0x7f57b0009b30).stop 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- 172.21.15.112:0/118477265 shutdown_connections 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 --2- 172.21.15.112:0/118477265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c00f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- 172.21.15.112:0/118477265 >> 172.21.15.112:0/118477265 conn(0x7f57c00f0b70 msgr2=0x7f57c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- 172.21.15.112:0/118477265 shutdown_connections 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- 172.21.15.112:0/118477265 wait complete. 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 Processor -- start 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- start start 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.284+0000 7f57c6805700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0105bd0 con 0x7f57c00f5410 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:59662/0 (socket says 172.21.15.112:59662) 2024-02-13T03:28:21.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bffff700 1 -- 172.21.15.112:0/413741582 learned_addr learned my addr 172.21.15.112:0/413741582 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bffff700 1 -- 172.21.15.112:0/413741582 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0005040 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bffff700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f57b0005010 tx=0x7f57b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b0009320 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b0013380 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c0105dd0 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b0014e50 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.285+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c0106260 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.286+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57b0014420 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.286+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c0045ad0 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.286+0000 7f57bcff9700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 0x7f57a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.286+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57b0040eb0 con 0x7f57c00f5410 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.286+0000 7f57bf7fe700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 0x7f57a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:21.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.287+0000 7f57bf7fe700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 0x7f57a80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f57b40096a0 tx=0x7f57b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:21.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.291+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57b00095f0 con 0x7f57c00f5410 2024-02-13T03:28:21.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.498+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57c0107110 con 0x7f57c00f5410 2024-02-13T03:28:21.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.498+0000 7f57bcff9700 1 -- 172.21.15.112:0/413741582 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57b001db70 con 0x7f57c00f5410 2024-02-13T03:28:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:21.499 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 msgr2=0x7f57a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 0x7f57a80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f57b40096a0 tx=0x7f57b4006b40).stop 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 msgr2=0x7f57c01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f57b0005010 tx=0x7f57b000b040).stop 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 shutdown_connections 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f57a802ff10 0x7f57a80323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 --2- 172.21.15.112:0/413741582 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57c00f5410 0x7f57c01056e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 >> 172.21.15.112:0/413741582 conn(0x7f57c00f0b70 msgr2=0x7f57c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:21.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 shutdown_connections 2024-02-13T03:28:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:21.500+0000 7f57c6805700 1 -- 172.21.15.112:0/413741582 wait complete. 2024-02-13T03:28:21.502 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:22.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:20.733823+0000 mgr.smithi080.vyzlfg (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:21.499635+0000 mon.smithi080 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.112:0/413741582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:23.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:23.153 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:24.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- 172.21.15.112:0/1760362757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 msgr2=0x7f567c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:24.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 --2- 172.21.15.112:0/1760362757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c0f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5670009fb0 tx=0x7f5670009b30).stop 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- 172.21.15.112:0/1760362757 shutdown_connections 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 --2- 172.21.15.112:0/1760362757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c0f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- 172.21.15.112:0/1760362757 >> 172.21.15.112:0/1760362757 conn(0x7f567c0f1c70 msgr2=0x7f567c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- 172.21.15.112:0/1760362757 shutdown_connections 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- 172.21.15.112:0/1760362757 wait complete. 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 Processor -- start 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.919+0000 7f5682bae700 1 -- start start 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f5682bae700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:24.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f5682bae700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c106cd0 con 0x7f567c0f6510 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f568094a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f568094a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52372/0 (socket says 172.21.15.112:52372) 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f568094a700 1 -- 172.21.15.112:0/1919602841 learned_addr learned my addr 172.21.15.112:0/1919602841 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.920+0000 7f568094a700 1 -- 172.21.15.112:0/1919602841 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5670005040 con 0x7f567c0f6510 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f568094a700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5670005010 tx=0x7f567000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5670009320 con 0x7f567c0f6510 2024-02-13T03:28:24.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5670013380 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c106ed0 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5670014e50 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c107360 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.921+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56700263a0 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.922+0000 7f56797fa700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 0x7f566c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.922+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c045ad0 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.922+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5670040ee0 con 0x7f567c0f6510 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.922+0000 7f567bfff700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 0x7f566c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:24.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.922+0000 7f567bfff700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 0x7f566c0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f56680096a0 tx=0x7f5668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:24.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:24.926+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f567001dd70 con 0x7f567c0f6510 2024-02-13T03:28:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:22.734157+0000 mgr.smithi080.vyzlfg (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:25.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.132+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f567c108210 con 0x7f567c0f6510 2024-02-13T03:28:25.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.133+0000 7f56797fa700 1 -- 172.21.15.112:0/1919602841 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f567001d980 con 0x7f567c0f6510 2024-02-13T03:28:25.134 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:25.135 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:25.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 msgr2=0x7f566c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 0x7f566c0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f56680096a0 tx=0x7f5668006b40).stop 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 msgr2=0x7f567c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5670005010 tx=0x7f567000b040).stop 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 shutdown_connections 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566c02ff10 0x7f566c0323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:25.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 --2- 172.21.15.112:0/1919602841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f567c0f6510 0x7f567c1067e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:25.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 >> 172.21.15.112:0/1919602841 conn(0x7f567c0f1c70 msgr2=0x7f567c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:25.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 shutdown_connections 2024-02-13T03:28:25.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:25.134+0000 7f5682bae700 1 -- 172.21.15.112:0/1919602841 wait complete. 2024-02-13T03:28:25.138 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:25.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:25 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:25.134254+0000 mon.smithi080 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.112:0/1919602841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:26.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:26.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:24.734362+0000 mgr.smithi080.vyzlfg (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.510+0000 7f2fe139f700 1 -- 172.21.15.112:0/1679136255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 msgr2=0x7f2fdc0f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.510+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1679136255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc0f5c30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2fcc009fb0 tx=0x7f2fcc009b30).stop 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.510+0000 7f2fe139f700 1 -- 172.21.15.112:0/1679136255 shutdown_connections 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.510+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1679136255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc0f5c30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.511+0000 7f2fe139f700 1 -- 172.21.15.112:0/1679136255 >> 172.21.15.112:0/1679136255 conn(0x7f2fdc0f0120 msgr2=0x7f2fdc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.511+0000 7f2fe139f700 1 -- 172.21.15.112:0/1679136255 shutdown_connections 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.511+0000 7f2fe139f700 1 -- 172.21.15.112:0/1679136255 wait complete. 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.511+0000 7f2fe139f700 1 Processor -- start 2024-02-13T03:28:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.511+0000 7f2fe139f700 1 -- start start 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fe139f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fe139f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc105960 con 0x7f2fdc0f5810 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fdbfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fdbfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52390/0 (socket says 172.21.15.112:52390) 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fdbfff700 1 -- 172.21.15.112:0/1362700008 learned_addr learned my addr 172.21.15.112:0/1362700008 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fdbfff700 1 -- 172.21.15.112:0/1362700008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc005040 con 0x7f2fdc0f5810 2024-02-13T03:28:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.512+0000 7f2fdbfff700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2fcc009830 tx=0x7f2fcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fcc018070 con 0x7f2fdc0f5810 2024-02-13T03:28:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc105b60 con 0x7f2fdc0f5810 2024-02-13T03:28:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fcc013410 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc105ff0 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fcc01c3c0 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fcc01d420 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc045ad0 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.513+0000 7f2fd8ff9700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 0x7f2fc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.514+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fcc041440 con 0x7f2fdc0f5810 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.514+0000 7f2fdb7fe700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 0x7f2fc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.514+0000 7f2fdb7fe700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 0x7f2fc4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.518+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fcc013ca0 con 0x7f2fdc0f5810 2024-02-13T03:28:28.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.723+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fdc106ea0 con 0x7f2fdc0f5810 2024-02-13T03:28:28.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.724+0000 7f2fd8ff9700 1 -- 172.21.15.112:0/1362700008 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fcc027e60 con 0x7f2fdc0f5810 2024-02-13T03:28:28.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:28.725 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 msgr2=0x7f2fc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 0x7f2fc4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).stop 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 msgr2=0x7f2fdc105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2fcc009830 tx=0x7f2fcc009b30).stop 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 shutdown_connections 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2fc402ff60 0x7f2fc4032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 --2- 172.21.15.112:0/1362700008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc0f5810 0x7f2fdc105470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.725+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 >> 172.21.15.112:0/1362700008 conn(0x7f2fdc0f0120 msgr2=0x7f2fdc0f0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.726+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 shutdown_connections 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:28.726+0000 7f2fe139f700 1 -- 172.21.15.112:0/1362700008 wait complete. 2024-02-13T03:28:28.727 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:26.734649+0000 mgr.smithi080.vyzlfg (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:29.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:29 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:28.725336+0000 mon.smithi080 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.112:0/1362700008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:30.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:30.362 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:28.734976+0000 mgr.smithi080.vyzlfg (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:32.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 -- 172.21.15.112:0/2872268019 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c0f6510 msgr2=0x7fb55c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 --2- 172.21.15.112:0/2872268019 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c0f6510 0x7fb55c0f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb550009fb0 tx=0x7fb550009b30).stop 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 -- 172.21.15.112:0/2872268019 shutdown_connections 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 --2- 172.21.15.112:0/2872268019 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c0f6510 0x7fb55c0f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 -- 172.21.15.112:0/2872268019 >> 172.21.15.112:0/2872268019 conn(0x7fb55c0f1c70 msgr2=0x7fb55c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 -- 172.21.15.112:0/2872268019 shutdown_connections 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.309+0000 7fb5619b6700 1 -- 172.21.15.112:0/2872268019 wait complete. 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb5619b6700 1 Processor -- start 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb5619b6700 1 -- start start 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb5619b6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb5619b6700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c107210 con 0x7fb55c106900 2024-02-13T03:28:32.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb55affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:32.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb55affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33546/0 (socket says 172.21.15.112:33546) 2024-02-13T03:28:32.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb55affd700 1 -- 172.21.15.112:0/3993720346 learned_addr learned my addr 172.21.15.112:0/3993720346 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:32.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.310+0000 7fb55affd700 1 -- 172.21.15.112:0/3993720346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550005040 con 0x7fb55c106900 2024-02-13T03:28:32.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb55affd700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb55c0f7610 tx=0x7fb550009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb550013380 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb550013790 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c107410 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb55001ee70 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.311+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c107f60 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb550025060 con 0x7fb55c106900 2024-02-13T03:28:32.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c059100 con 0x7fb55c106900 2024-02-13T03:28:32.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb5437fe700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:32.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb550040e60 con 0x7fb55c106900 2024-02-13T03:28:32.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb55a7fc700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:32.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.312+0000 7fb55a7fc700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 0x7fb544032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:32.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.316+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb55001c950 con 0x7fb55c106900 2024-02-13T03:28:32.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.519+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb55c045ad0 con 0x7fb55c106900 2024-02-13T03:28:32.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.520+0000 7fb5437fe700 1 -- 172.21.15.112:0/3993720346 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb55002d2d0 con 0x7fb55c106900 2024-02-13T03:28:32.521 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:32.521 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 msgr2=0x7fb544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 0x7fb544032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).stop 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 msgr2=0x7fb55c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb55c0f7610 tx=0x7fb550009b30).stop 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 shutdown_connections 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:32.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 --2- 172.21.15.112:0/3993720346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb55c106900 0x7fb55c106d20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:32.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 >> 172.21.15.112:0/3993720346 conn(0x7fb55c0f1c70 msgr2=0x7fb55c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:32.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 shutdown_connections 2024-02-13T03:28:32.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:32.521+0000 7fb5619b6700 1 -- 172.21.15.112:0/3993720346 wait complete. 2024-02-13T03:28:32.523 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:30.735318+0000 mgr.smithi080.vyzlfg (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:33.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:32.521223+0000 mon.smithi080 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.112:0/3993720346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:34.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:34.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:32.735608+0000 mgr.smithi080.vyzlfg (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:35.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.837+0000 7f77bd461700 1 -- 172.21.15.112:0/1506224566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 msgr2=0x7f77b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.837+0000 7f77bd461700 1 --2- 172.21.15.112:0/1506224566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b80f66c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f77a8009fb0 tx=0x7f77a8009b30).stop 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.837+0000 7f77bd461700 1 -- 172.21.15.112:0/1506224566 shutdown_connections 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.837+0000 7f77bd461700 1 --2- 172.21.15.112:0/1506224566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b80f66c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.837+0000 7f77bd461700 1 -- 172.21.15.112:0/1506224566 >> 172.21.15.112:0/1506224566 conn(0x7f77b80f1ac0 msgr2=0x7f77b80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 -- 172.21.15.112:0/1506224566 shutdown_connections 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 -- 172.21.15.112:0/1506224566 wait complete. 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 Processor -- start 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 -- start start 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77bd461700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b8106a60 con 0x7f77b80f62a0 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77b7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77b7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33562/0 (socket says 172.21.15.112:33562) 2024-02-13T03:28:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.838+0000 7f77b7fff700 1 -- 172.21.15.112:0/2627850245 learned_addr learned my addr 172.21.15.112:0/2627850245 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77b7fff700 1 -- 172.21.15.112:0/2627850245 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a8005040 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77b7fff700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f77a80089f0 tx=0x7f77a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a8009320 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77a8013380 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a8014e20 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b8106c60 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.839+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b81070f0 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.840+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77a8014420 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.840+0000 7f77b4ff9700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 0x7f77a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.840+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77a8040f00 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.840+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b8045ad0 con 0x7f77b80f62a0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.840+0000 7f77b77fe700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 0x7f77a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.841+0000 7f77b77fe700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 0x7f77a0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f77ac0096a0 tx=0x7f77ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:35.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:35.845+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77a80095f0 con 0x7f77b80f62a0 2024-02-13T03:28:36.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.048+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77b8107fa0 con 0x7f77b80f62a0 2024-02-13T03:28:36.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.048+0000 7f77b4ff9700 1 -- 172.21.15.112:0/2627850245 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77a801db70 con 0x7f77b80f62a0 2024-02-13T03:28:36.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:36.049 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 msgr2=0x7f77a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 0x7f77a0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f77ac0096a0 tx=0x7f77ac006b40).stop 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 msgr2=0x7f77b8106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f77a80089f0 tx=0x7f77a800b040).stop 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 shutdown_connections 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f77a002ff60 0x7f77a0032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 --2- 172.21.15.112:0/2627850245 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f77b80f62a0 0x7f77b8106570 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 >> 172.21.15.112:0/2627850245 conn(0x7f77b80f1ac0 msgr2=0x7f77b80f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:36.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 shutdown_connections 2024-02-13T03:28:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:36.050+0000 7f77bd461700 1 -- 172.21.15.112:0/2627850245 wait complete. 2024-02-13T03:28:36.052 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:36.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:34.735937+0000 mgr.smithi080.vyzlfg (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:36.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:36.049626+0000 mon.smithi080 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.112:0/2627850245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:37.766 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:37.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:38.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:36.736308+0000 mgr.smithi080.vyzlfg (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:39.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 -- 172.21.15.112:0/1582578904 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 msgr2=0x7f50180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:39.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 --2- 172.21.15.112:0/1582578904 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50180f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5008009fb0 tx=0x7f5008009b30).stop 2024-02-13T03:28:39.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 -- 172.21.15.112:0/1582578904 shutdown_connections 2024-02-13T03:28:39.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 --2- 172.21.15.112:0/1582578904 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50180f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:39.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 -- 172.21.15.112:0/1582578904 >> 172.21.15.112:0/1582578904 conn(0x7f50180f0b70 msgr2=0x7f50180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.586+0000 7f501e1aa700 1 -- 172.21.15.112:0/1582578904 shutdown_connections 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.587+0000 7f501e1aa700 1 -- 172.21.15.112:0/1582578904 wait complete. 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.587+0000 7f501e1aa700 1 Processor -- start 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.587+0000 7f501e1aa700 1 -- start start 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f501e1aa700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f501e1aa700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018105bd0 con 0x7f50180f5410 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f50177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f50177fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33580/0 (socket says 172.21.15.112:33580) 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f50177fe700 1 -- 172.21.15.112:0/2313930924 learned_addr learned my addr 172.21.15.112:0/2313930924 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f50177fe700 1 -- 172.21.15.112:0/2313930924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008005040 con 0x7f50180f5410 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f50177fe700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5008005010 tx=0x7f500800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:39.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5008009320 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5008013380 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5018105dd0 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5008014e50 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.588+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5018106260 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.589+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50080095f0 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.589+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5018045ad0 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.589+0000 7f500ffff700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.590+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5008040e70 con 0x7f50180f5410 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.590+0000 7f5016ffd700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.590+0000 7f5016ffd700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f50000096a0 tx=0x7f5000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:39.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.594+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5008014420 con 0x7f50180f5410 2024-02-13T03:28:39.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.799+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5018107110 con 0x7f50180f5410 2024-02-13T03:28:39.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.800+0000 7f500ffff700 1 -- 172.21.15.112:0/2313930924 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5008010230 con 0x7f50180f5410 2024-02-13T03:28:39.799 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:39.800 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 msgr2=0x7f50040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f50000096a0 tx=0x7f5000006b40).stop 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 msgr2=0x7f50181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5008005010 tx=0x7f500800b040).stop 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 shutdown_connections 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 --2- 172.21.15.112:0/2313930924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f5410 0x7f50181056e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 >> 172.21.15.112:0/2313930924 conn(0x7f50180f0b70 msgr2=0x7f50180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:39.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 shutdown_connections 2024-02-13T03:28:39.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:39.801+0000 7f501e1aa700 1 -- 172.21.15.112:0/2313930924 wait complete. 2024-02-13T03:28:39.802 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:40.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:38.736703+0000 mgr.smithi080.vyzlfg (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:40.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:39.799996+0000 mon.smithi080 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.112:0/2313930924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:41.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:41.484 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:40.737117+0000 mgr.smithi080.vyzlfg (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:43.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.289+0000 7fe7ea04e700 1 -- 172.21.15.112:0/450397870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 msgr2=0x7fe7e40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.289+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/450397870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e40f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe7e0009fb0 tx=0x7fe7e0009b30).stop 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- 172.21.15.112:0/450397870 shutdown_connections 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/450397870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e40f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- 172.21.15.112:0/450397870 >> 172.21.15.112:0/450397870 conn(0x7fe7e40f1ac0 msgr2=0x7fe7e40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- 172.21.15.112:0/450397870 shutdown_connections 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- 172.21.15.112:0/450397870 wait complete. 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 Processor -- start 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- start start 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:43.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7ea04e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4069e20 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7e904c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7e904c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53634/0 (socket says 172.21.15.112:53634) 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.290+0000 7fe7e904c700 1 -- 172.21.15.112:0/1004004123 learned_addr learned my addr 172.21.15.112:0/1004004123 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7e904c700 1 -- 172.21.15.112:0/1004004123 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0005040 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7e904c700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe7e00089f0 tx=0x7fe7e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e0009320 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e406a020 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7e0013380 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e0014e20 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.291+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e406a4b0 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.292+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7e0014420 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.292+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e4059100 con 0x7fe7e40f62a0 2024-02-13T03:28:43.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.292+0000 7fe7d9ffb700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 0x7fe7d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.292+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7e0040fc0 con 0x7fe7e40f62a0 2024-02-13T03:28:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.292+0000 7fe7e884b700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 0x7fe7d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.293+0000 7fe7e884b700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 0x7fe7d00323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fe7d40096a0 tx=0x7fe7d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:43.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.297+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7e006d2b0 con 0x7fe7e40f62a0 2024-02-13T03:28:43.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.505+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7e40fbe30 con 0x7fe7e40f62a0 2024-02-13T03:28:43.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.505+0000 7fe7d9ffb700 1 -- 172.21.15.112:0/1004004123 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7e00095f0 con 0x7fe7e40f62a0 2024-02-13T03:28:43.505 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:43.506 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 msgr2=0x7fe7d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 0x7fe7d00323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fe7d40096a0 tx=0x7fe7d4006b40).stop 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 msgr2=0x7fe7e406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe7e00089f0 tx=0x7fe7e000b040).stop 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 shutdown_connections 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe7d002ff10 0x7fe7d00323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 --2- 172.21.15.112:0/1004004123 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7e40f62a0 0x7fe7e406b930 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:43.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 >> 172.21.15.112:0/1004004123 conn(0x7fe7e40f1ac0 msgr2=0x7fe7e40f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:43.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 shutdown_connections 2024-02-13T03:28:43.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:43.507+0000 7fe7ea04e700 1 -- 172.21.15.112:0/1004004123 wait complete. 2024-02-13T03:28:43.508 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:42.737515+0000 mgr.smithi080.vyzlfg (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:44.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:43.505730+0000 mon.smithi080 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.112:0/1004004123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:45.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:45.077 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:46.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 -- 172.21.15.112:0/3488480441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 msgr2=0x7f36cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/3488480441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc0f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f36c4009960 tx=0x7f36c4004d10).stop 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 -- 172.21.15.112:0/3488480441 shutdown_connections 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/3488480441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc0f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 -- 172.21.15.112:0/3488480441 >> 172.21.15.112:0/3488480441 conn(0x7f36cc0f1c70 msgr2=0x7f36cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 -- 172.21.15.112:0/3488480441 shutdown_connections 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.809+0000 7f36d0cd8700 1 -- 172.21.15.112:0/3488480441 wait complete. 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36d0cd8700 1 Processor -- start 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36d0cd8700 1 -- start start 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36d0cd8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36d0cd8700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc069e20 con 0x7f36cc0f6510 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36ca59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36ca59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53660/0 (socket says 172.21.15.112:53660) 2024-02-13T03:28:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36ca59c700 1 -- 172.21.15.112:0/2178411914 learned_addr learned my addr 172.21.15.112:0/2178411914 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36ca59c700 1 -- 172.21.15.112:0/2178411914 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4009640 con 0x7f36cc0f6510 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.810+0000 7f36ca59c700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f36c4004a10 tx=0x7f36c4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c4018070 con 0x7f36cc0f6510 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc069fc0 con 0x7f36cc0f6510 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c4007e70 con 0x7f36cc0f6510 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c401d420 con 0x7f36cc0f6510 2024-02-13T03:28:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc06a450 con 0x7f36cc0f6510 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36c401d6e0 con 0x7f36cc0f6510 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.811+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc059100 con 0x7f36cc0f6510 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.812+0000 7f36baffd700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 0x7f36b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.812+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36c4041910 con 0x7f36cc0f6510 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.812+0000 7f36c9d9b700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 0x7f36b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:46.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.812+0000 7f36c9d9b700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 0x7f36b4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f36c00096a0 tx=0x7f36c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:46.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:46.816+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36c401dae0 con 0x7f36cc0f6510 2024-02-13T03:28:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:44.737852+0000 mgr.smithi080.vyzlfg (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.019+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36cc045ad0 con 0x7f36cc0f6510 2024-02-13T03:28:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.020+0000 7f36baffd700 1 -- 172.21.15.112:0/2178411914 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36c4011320 con 0x7f36cc0f6510 2024-02-13T03:28:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:47.021 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 msgr2=0x7f36b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 0x7f36b4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f36c00096a0 tx=0x7f36c0006b40).stop 2024-02-13T03:28:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 msgr2=0x7f36cc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f36c4004a10 tx=0x7f36c4004d10).stop 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 shutdown_connections 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f36b402ff60 0x7f36b4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 --2- 172.21.15.112:0/2178411914 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f36cc0f6510 0x7f36cc06b9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 >> 172.21.15.112:0/2178411914 conn(0x7f36cc0f1c70 msgr2=0x7f36cc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 shutdown_connections 2024-02-13T03:28:47.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:47.021+0000 7f36d0cd8700 1 -- 172.21.15.112:0/2178411914 wait complete. 2024-02-13T03:28:47.025 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:47.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:47.021091+0000 mon.smithi080 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.112:0/2178411914' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:48.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:48.633 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:46.738244+0000 mgr.smithi080.vyzlfg (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:50.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 -- 172.21.15.112:0/462998906 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 msgr2=0x7f4da00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 --2- 172.21.15.112:0/462998906 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da00f90f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f4d90009fb0 tx=0x7f4d90009b30).stop 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 -- 172.21.15.112:0/462998906 shutdown_connections 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 --2- 172.21.15.112:0/462998906 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da00f90f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 -- 172.21.15.112:0/462998906 >> 172.21.15.112:0/462998906 conn(0x7f4da00f0b70 msgr2=0x7f4da00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 -- 172.21.15.112:0/462998906 shutdown_connections 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.502+0000 7f4da5d91700 1 -- 172.21.15.112:0/462998906 wait complete. 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4da5d91700 1 Processor -- start 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4da5d91700 1 -- start start 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4da5d91700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4da5d91700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0108e80 con 0x7f4da0060280 2024-02-13T03:28:50.505 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4d9f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4d9f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53686/0 (socket says 172.21.15.112:53686) 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4d9f7fe700 1 -- 172.21.15.112:0/70581910 learned_addr learned my addr 172.21.15.112:0/70581910 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.503+0000 7f4d9f7fe700 1 -- 172.21.15.112:0/70581910 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d90005040 con 0x7f4da0060280 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4d9f7fe700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f4d90009830 tx=0x7f4d9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d90009320 con 0x7f4da0060280 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da0109080 con 0x7f4da0060280 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d90013380 con 0x7f4da0060280 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d90014420 con 0x7f4da0060280 2024-02-13T03:28:50.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da0109510 con 0x7f4da0060280 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.504+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d90014580 con 0x7f4da0060280 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.505+0000 7f4d97fff700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 0x7f4d8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.505+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d90040e60 con 0x7f4da0060280 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.505+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da0045ad0 con 0x7f4da0060280 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.505+0000 7f4d9effd700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 0x7f4d8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:50.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.505+0000 7f4d9effd700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 0x7f4d8c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4d880096a0 tx=0x7f4d88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:50.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.509+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d900273f0 con 0x7f4da0060280 2024-02-13T03:28:50.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.716+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4da00fc090 con 0x7f4da0060280 2024-02-13T03:28:50.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.717+0000 7f4d97fff700 1 -- 172.21.15.112:0/70581910 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d9001c890 con 0x7f4da0060280 2024-02-13T03:28:50.718 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:50.718 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:50.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 msgr2=0x7f4d8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:50.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 0x7f4d8c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4d880096a0 tx=0x7f4d88006b40).stop 2024-02-13T03:28:50.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 msgr2=0x7f4da0108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f4d90009830 tx=0x7f4d9000b040).stop 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 shutdown_connections 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4d8c02ff60 0x7f4d8c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 --2- 172.21.15.112:0/70581910 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4da0060280 0x7f4da0108990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 >> 172.21.15.112:0/70581910 conn(0x7f4da00f0b70 msgr2=0x7f4da00f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 shutdown_connections 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:50.718+0000 7f4da5d91700 1 -- 172.21.15.112:0/70581910 wait complete. 2024-02-13T03:28:50.720 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:48.738541+0000 mgr.smithi080.vyzlfg (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:51.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:50.718104+0000 mon.smithi080 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.112:0/70581910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:52.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:52.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:50.738839+0000 mgr.smithi080.vyzlfg (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:54.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.175+0000 7f5064144700 1 -- 172.21.15.112:0/633430140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 msgr2=0x7f505c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.175+0000 7f5064144700 1 --2- 172.21.15.112:0/633430140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f504c009fb0 tx=0x7f504c009b30).stop 2024-02-13T03:28:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.175+0000 7f5064144700 1 -- 172.21.15.112:0/633430140 shutdown_connections 2024-02-13T03:28:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.175+0000 7f5064144700 1 --2- 172.21.15.112:0/633430140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.175+0000 7f5064144700 1 -- 172.21.15.112:0/633430140 >> 172.21.15.112:0/633430140 conn(0x7f505c0f0b70 msgr2=0x7f505c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 -- 172.21.15.112:0/633430140 shutdown_connections 2024-02-13T03:28:54.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 -- 172.21.15.112:0/633430140 wait complete. 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 Processor -- start 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 -- start start 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.176+0000 7f5064144700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c060240 con 0x7f505c0f5410 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5061ee0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5061ee0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:51692/0 (socket says 172.21.15.112:51692) 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5061ee0700 1 -- 172.21.15.112:0/2624863047 learned_addr learned my addr 172.21.15.112:0/2624863047 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5061ee0700 1 -- 172.21.15.112:0/2624863047 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c005040 con 0x7f505c0f5410 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5061ee0700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f504c005010 tx=0x7f504c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f504c009320 con 0x7f505c0f5410 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c060440 con 0x7f505c0f5410 2024-02-13T03:28:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f504c013380 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f504c014e50 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.177+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c109b90 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.178+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f504c0263a0 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.178+0000 7f50527fc700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 0x7f50480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.178+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f504c040fb0 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.178+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505c045ad0 con 0x7f505c0f5410 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.179+0000 7f50616df700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 0x7f50480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.179+0000 7f50616df700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 0x7f50480323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f50580096a0 tx=0x7f5058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.184+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f504c012920 con 0x7f505c0f5410 2024-02-13T03:28:54.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.392+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f505c0fafa0 con 0x7f505c0f5410 2024-02-13T03:28:54.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.392+0000 7f50527fc700 1 -- 172.21.15.112:0/2624863047 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f504c012330 con 0x7f505c0f5410 2024-02-13T03:28:54.394 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:54.394 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:54.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 msgr2=0x7f50480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 0x7f50480323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f50580096a0 tx=0x7f5058006b40).stop 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 msgr2=0x7f505c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f504c005010 tx=0x7f504c00b040).stop 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 shutdown_connections 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f504802ff10 0x7f50480323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 --2- 172.21.15.112:0/2624863047 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f505c0f5410 0x7f505c1094e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 >> 172.21.15.112:0/2624863047 conn(0x7f505c0f0b70 msgr2=0x7f505c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 shutdown_connections 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:54.394+0000 7f5064144700 1 -- 172.21.15.112:0/2624863047 wait complete. 2024-02-13T03:28:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:52.739175+0000 mgr.smithi080.vyzlfg (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:54.393563+0000 mon.smithi080 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.112:0/2624863047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:55.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:55.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:28:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:54.739495+0000 mgr.smithi080.vyzlfg (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:57.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- 172.21.15.112:0/1676899281 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 msgr2=0x7f5a5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:57.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 --2- 172.21.15.112:0/1676899281 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c0f6930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5a50009fb0 tx=0x7f5a50009b30).stop 2024-02-13T03:28:57.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- 172.21.15.112:0/1676899281 shutdown_connections 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 --2- 172.21.15.112:0/1676899281 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c0f6930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- 172.21.15.112:0/1676899281 >> 172.21.15.112:0/1676899281 conn(0x7f5a5c0f1c70 msgr2=0x7f5a5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- 172.21.15.112:0/1676899281 shutdown_connections 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- 172.21.15.112:0/1676899281 wait complete. 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 Processor -- start 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.808+0000 7f5a610f5700 1 -- start start 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a610f5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a610f5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c106cd0 con 0x7f5a5c0f6510 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a5ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a5ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:51716/0 (socket says 172.21.15.112:51716) 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a5ad9d700 1 -- 172.21.15.112:0/374831950 learned_addr learned my addr 172.21.15.112:0/374831950 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a5ad9d700 1 -- 172.21.15.112:0/374831950 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a50005040 con 0x7f5a5c0f6510 2024-02-13T03:28:57.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a5ad9d700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5a50005010 tx=0x7f5a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:57.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.809+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a50009320 con 0x7f5a5c0f6510 2024-02-13T03:28:57.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c106ed0 con 0x7f5a5c0f6510 2024-02-13T03:28:57.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a50013380 con 0x7f5a5c0f6510 2024-02-13T03:28:57.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a50014e50 con 0x7f5a5c0f6510 2024-02-13T03:28:57.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c107360 con 0x7f5a5c0f6510 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a500095f0 con 0x7f5a5c0f6510 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.810+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c045ad0 con 0x7f5a5c0f6510 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.811+0000 7f5a437fe700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 0x7f5a440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.811+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a50040e90 con 0x7f5a5c0f6510 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.811+0000 7f5a5a59c700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 0x7f5a440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:28:57.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.811+0000 7f5a5a59c700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 0x7f5a440323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5a4c0096a0 tx=0x7f5a4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:28:57.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:57.815+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a50014420 con 0x7f5a5c0f6510 2024-02-13T03:28:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:56.854595+0000 mon.smithi080 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:28:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:56.862929+0000 mon.smithi080 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:28:58.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.020+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a5c108210 con 0x7f5a5c0f6510 2024-02-13T03:28:58.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.021+0000 7f5a437fe700 1 -- 172.21.15.112:0/374831950 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a50010230 con 0x7f5a5c0f6510 2024-02-13T03:28:58.022 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:28:58.022 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 msgr2=0x7f5a440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 0x7f5a440323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5a4c0096a0 tx=0x7f5a4c006b40).stop 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 msgr2=0x7f5a5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5a50005010 tx=0x7f5a5000b040).stop 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 shutdown_connections 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5a4402ff10 0x7f5a440323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 --2- 172.21.15.112:0/374831950 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a5c0f6510 0x7f5a5c1067e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:28:58.024 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 >> 172.21.15.112:0/374831950 conn(0x7f5a5c0f1c70 msgr2=0x7f5a5c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:28:58.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 shutdown_connections 2024-02-13T03:28:58.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:28:58.023+0000 7f5a610f5700 1 -- 172.21.15.112:0/374831950 wait complete. 2024-02-13T03:28:58.025 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:28:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:56.739765+0000 mgr.smithi080.vyzlfg (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:28:58.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:28:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:28:58.022377+0000 mon.smithi080 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.112:0/374831950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:28:59.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:28:59.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:28:58.740037+0000 mgr.smithi080.vyzlfg (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:01.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 -- 172.21.15.112:0/2211662777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 msgr2=0x7f47c00f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2211662777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00f57a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f47a8009fb0 tx=0x7f47a8009b30).stop 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 -- 172.21.15.112:0/2211662777 shutdown_connections 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2211662777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00f57a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 -- 172.21.15.112:0/2211662777 >> 172.21.15.112:0/2211662777 conn(0x7f47c00f0b00 msgr2=0x7f47c00f2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 -- 172.21.15.112:0/2211662777 shutdown_connections 2024-02-13T03:29:01.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.511+0000 7f47c53cf700 1 -- 172.21.15.112:0/2211662777 wait complete. 2024-02-13T03:29:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47c53cf700 1 Processor -- start 2024-02-13T03:29:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47c53cf700 1 -- start start 2024-02-13T03:29:01.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47c53cf700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47c53cf700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00fb950 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47beffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47328/0 (socket says 172.21.15.112:47328) 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.512+0000 7f47beffd700 1 -- 172.21.15.112:0/2007963695 learned_addr learned my addr 172.21.15.112:0/2007963695 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47beffd700 1 -- 172.21.15.112:0/2007963695 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8005040 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47beffd700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f47a8005010 tx=0x7f47a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a8009320 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c00fbaf0 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c00fbf80 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a8013380 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.513+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a8014d10 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.514+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4004fa0 con 0x7f47c00f5380 2024-02-13T03:29:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.514+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47a8014420 con 0x7f47c00f5380 2024-02-13T03:29:01.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.514+0000 7f47b77fe700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 0x7f47ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:01.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.514+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47a8040da0 con 0x7f47c00f5380 2024-02-13T03:29:01.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.514+0000 7f47be7fc700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 0x7f47ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:01.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.515+0000 7f47be7fc700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 0x7f47ac032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f47b00096a0 tx=0x7f47b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:01.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.519+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47a8004490 con 0x7f47c00f5380 2024-02-13T03:29:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.722+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a4005670 con 0x7f47c00f5380 2024-02-13T03:29:01.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.723+0000 7f47b77fe700 1 -- 172.21.15.112:0/2007963695 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47a8012750 con 0x7f47c00f5380 2024-02-13T03:29:01.724 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:01.724 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 msgr2=0x7f47ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 0x7f47ac032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f47b00096a0 tx=0x7f47b0006b40).stop 2024-02-13T03:29:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 msgr2=0x7f47c00fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f47a8005010 tx=0x7f47a800b040).stop 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 shutdown_connections 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f47ac02ff60 0x7f47ac032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 --2- 172.21.15.112:0/2007963695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f47c00f5380 0x7f47c00fd4d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 >> 172.21.15.112:0/2007963695 conn(0x7f47c00f0b00 msgr2=0x7f47c005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 shutdown_connections 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:01.724+0000 7f47c53cf700 1 -- 172.21.15.112:0/2007963695 wait complete. 2024-02-13T03:29:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:02.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:00.740419+0000 mgr.smithi080.vyzlfg (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:02.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:01.724263+0000 mon.smithi080 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.112:0/2007963695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:03.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:03.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:02.740682+0000 mgr.smithi080.vyzlfg (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 -- 172.21.15.112:0/610451271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 msgr2=0x7fd98c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 --2- 172.21.15.112:0/610451271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd974009fb0 tx=0x7fd974009b30).stop 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 -- 172.21.15.112:0/610451271 shutdown_connections 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 --2- 172.21.15.112:0/610451271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 -- 172.21.15.112:0/610451271 >> 172.21.15.112:0/610451271 conn(0x7fd98c0f0b70 msgr2=0x7fd98c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 -- 172.21.15.112:0/610451271 shutdown_connections 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.206+0000 7fd99299e700 1 -- 172.21.15.112:0/610451271 wait complete. 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd99299e700 1 Processor -- start 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd99299e700 1 -- start start 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd99299e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd99299e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c060220 con 0x7fd98c0f5410 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd98bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:05.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd98bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47338/0 (socket says 172.21.15.112:47338) 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd98bfff700 1 -- 172.21.15.112:0/3286285497 learned_addr learned my addr 172.21.15.112:0/3286285497 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.207+0000 7fd98bfff700 1 -- 172.21.15.112:0/3286285497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974005040 con 0x7fd98c0f5410 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd98bfff700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd9740089f0 tx=0x7fd97400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd974009320 con 0x7fd98c0f5410 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd974013380 con 0x7fd98c0f5410 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98c060420 con 0x7fd98c0f5410 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd97401ed80 con 0x7fd98c0f5410 2024-02-13T03:29:05.210 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.208+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98c109bb0 con 0x7fd98c0f5410 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd97401e420 con 0x7fd98c0f5410 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98c059100 con 0x7fd98c0f5410 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd988ff9700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 0x7fd978032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9740410c0 con 0x7fd98c0f5410 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd98b7fe700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 0x7fd978032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:05.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.209+0000 7fd98b7fe700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 0x7fd978032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd97c0096a0 tx=0x7fd97c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:05.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.214+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd97401cb10 con 0x7fd98c0f5410 2024-02-13T03:29:05.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.416+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd98c045ad0 con 0x7fd98c0f5410 2024-02-13T03:29:05.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.417+0000 7fd988ff9700 1 -- 172.21.15.112:0/3286285497 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd97401cb10 con 0x7fd98c0f5410 2024-02-13T03:29:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:05.418 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:05.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 msgr2=0x7fd978032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:05.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 0x7fd978032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd97c0096a0 tx=0x7fd97c006b40).stop 2024-02-13T03:29:05.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 msgr2=0x7fd98c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd9740089f0 tx=0x7fd97400b040).stop 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 shutdown_connections 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd97802ffb0 0x7fd978032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 --2- 172.21.15.112:0/3286285497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd98c0f5410 0x7fd98c109500 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 >> 172.21.15.112:0/3286285497 conn(0x7fd98c0f0b70 msgr2=0x7fd98c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 shutdown_connections 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:05.418+0000 7fd99299e700 1 -- 172.21.15.112:0/3286285497 wait complete. 2024-02-13T03:29:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:06.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:04.740933+0000 mgr.smithi080.vyzlfg (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:06.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:05.418252+0000 mon.smithi080 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.112:0/3286285497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:07.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:07.088 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:08.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:06.741195+0000 mgr.smithi080.vyzlfg (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 -- 172.21.15.112:0/1677691484 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 msgr2=0x7f93580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 --2- 172.21.15.112:0/1677691484 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93580f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f9348009fb0 tx=0x7f9348009b30).stop 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 -- 172.21.15.112:0/1677691484 shutdown_connections 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 --2- 172.21.15.112:0/1677691484 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93580f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 -- 172.21.15.112:0/1677691484 >> 172.21.15.112:0/1677691484 conn(0x7f93580f1c70 msgr2=0x7f93580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 -- 172.21.15.112:0/1677691484 shutdown_connections 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.054+0000 7f935dd49700 1 -- 172.21.15.112:0/1677691484 wait complete. 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f935dd49700 1 Processor -- start 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f935dd49700 1 -- start start 2024-02-13T03:29:09.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f935dd49700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:09.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f935dd49700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358106cd0 con 0x7f93580f6510 2024-02-13T03:29:09.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f93577fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:09.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f93577fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47352/0 (socket says 172.21.15.112:47352) 2024-02-13T03:29:09.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.055+0000 7f93577fe700 1 -- 172.21.15.112:0/3341652297 learned_addr learned my addr 172.21.15.112:0/3341652297 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:09.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f93577fe700 1 -- 172.21.15.112:0/3341652297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348005040 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f93577fe700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9348005010 tx=0x7f934800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9348009320 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9358106ed0 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9348013380 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9348014e50 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.056+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9358107360 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.057+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93480263a0 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.057+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9358045ad0 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.057+0000 7f934ffff700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 0x7f93440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.057+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9348041250 con 0x7f93580f6510 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.057+0000 7f9356ffd700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 0x7f93440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.058+0000 7f9356ffd700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 0x7f93440323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f93400096a0 tx=0x7f9340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:09.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.062+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93480126f0 con 0x7f93580f6510 2024-02-13T03:29:09.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.265+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9358108210 con 0x7f93580f6510 2024-02-13T03:29:09.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.267+0000 7f934ffff700 1 -- 172.21.15.112:0/3341652297 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9348012330 con 0x7f93580f6510 2024-02-13T03:29:09.267 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:09.267 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:09.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.267+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 msgr2=0x7f93440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.267+0000 7f935dd49700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 0x7f93440323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f93400096a0 tx=0x7f9340006b40).stop 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.267+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 msgr2=0x7f93581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.267+0000 7f935dd49700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9348005010 tx=0x7f934800b040).stop 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 shutdown_connections 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f934402ff10 0x7f93440323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 --2- 172.21.15.112:0/3341652297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f93580f6510 0x7f93581067e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 >> 172.21.15.112:0/3341652297 conn(0x7f93580f1c70 msgr2=0x7f93580f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 shutdown_connections 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:09.268+0000 7f935dd49700 1 -- 172.21.15.112:0/3341652297 wait complete. 2024-02-13T03:29:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:09.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:09.267438+0000 mon.smithi080 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.112:0/3341652297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:10.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:08.741504+0000 mgr.smithi080.vyzlfg (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:10.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:10.899 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:12.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:10.741802+0000 mgr.smithi080.vyzlfg (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 -- 172.21.15.112:0/3455947229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 msgr2=0x7f02c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 --2- 172.21.15.112:0/3455947229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c40f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f02b4009fb0 tx=0x7f02b4009b30).stop 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 -- 172.21.15.112:0/3455947229 shutdown_connections 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 --2- 172.21.15.112:0/3455947229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c40f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 -- 172.21.15.112:0/3455947229 >> 172.21.15.112:0/3455947229 conn(0x7f02c40f1c70 msgr2=0x7f02c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 -- 172.21.15.112:0/3455947229 shutdown_connections 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 -- 172.21.15.112:0/3455947229 wait complete. 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.739+0000 7f02c9cab700 1 Processor -- start 2024-02-13T03:29:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c9cab700 1 -- start start 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c9cab700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c9cab700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4069e20 con 0x7f02c40f6510 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:57074/0 (socket says 172.21.15.112:57074) 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c37fe700 1 -- 172.21.15.112:0/2557140448 learned_addr learned my addr 172.21.15.112:0/2557140448 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c37fe700 1 -- 172.21.15.112:0/2557140448 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b4005040 con 0x7f02c40f6510 2024-02-13T03:29:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.740+0000 7f02c37fe700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f02b4005010 tx=0x7f02b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b4009320 con 0x7f02c40f6510 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c4069fc0 con 0x7f02c40f6510 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02b4013380 con 0x7f02c40f6510 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b4014e50 con 0x7f02c40f6510 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c406a450 con 0x7f02c40f6510 2024-02-13T03:29:12.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.741+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02b40095f0 con 0x7f02c40f6510 2024-02-13T03:29:12.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.742+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c4059100 con 0x7f02c40f6510 2024-02-13T03:29:12.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.742+0000 7f02bbfff700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 0x7f02b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:12.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.742+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02b4040e90 con 0x7f02c40f6510 2024-02-13T03:29:12.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.742+0000 7f02c2ffd700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 0x7f02b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:12.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.742+0000 7f02c2ffd700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 0x7f02b00323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f02ac0096a0 tx=0x7f02ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:12.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.747+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02b4014420 con 0x7f02c40f6510 2024-02-13T03:29:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.951+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02c40fc0a0 con 0x7f02c40f6510 2024-02-13T03:29:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.953+0000 7f02bbfff700 1 -- 172.21.15.112:0/2557140448 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02b4010230 con 0x7f02c40f6510 2024-02-13T03:29:12.953 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:12.953 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 msgr2=0x7f02b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 0x7f02b00323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f02ac0096a0 tx=0x7f02ac006b40).stop 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 msgr2=0x7f02c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f02b4005010 tx=0x7f02b400b040).stop 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 shutdown_connections 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f02b002ff10 0x7f02b00323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 --2- 172.21.15.112:0/2557140448 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02c40f6510 0x7f02c406b9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 >> 172.21.15.112:0/2557140448 conn(0x7f02c40f1c70 msgr2=0x7f02c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 shutdown_connections 2024-02-13T03:29:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:12.954+0000 7f02c9cab700 1 -- 172.21.15.112:0/2557140448 wait complete. 2024-02-13T03:29:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:13.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:13 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:12.953374+0000 mon.smithi080 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.112:0/2557140448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:14.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:14.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:14.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:12.742039+0000 mgr.smithi080.vyzlfg (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 -- 172.21.15.112:0/2729262030 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 msgr2=0x7f3f2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 --2- 172.21.15.112:0/2729262030 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c0f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f3f28009fb0 tx=0x7f3f28009b30).stop 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 -- 172.21.15.112:0/2729262030 shutdown_connections 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 --2- 172.21.15.112:0/2729262030 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c0f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 -- 172.21.15.112:0/2729262030 >> 172.21.15.112:0/2729262030 conn(0x7f3f2c0f09a0 msgr2=0x7f3f2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 -- 172.21.15.112:0/2729262030 shutdown_connections 2024-02-13T03:29:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.482+0000 7f3f3347a700 1 -- 172.21.15.112:0/2729262030 wait complete. 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f3347a700 1 Processor -- start 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f3347a700 1 -- start start 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f3347a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f3347a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1059c0 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f32478700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f32478700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:57090/0 (socket says 172.21.15.112:57090) 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f32478700 1 -- 172.21.15.112:0/941215438 learned_addr learned my addr 172.21.15.112:0/941215438 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.483+0000 7f3f32478700 1 -- 172.21.15.112:0/941215438 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f28005040 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f32478700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f3f28009830 tx=0x7f3f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f28009320 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f28013380 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c105bc0 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f28014420 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.484+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c106050 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.485+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f28014580 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.485+0000 7f3f22ffd700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 0x7f3f18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:16.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.485+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f28040e20 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.485+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c045ad0 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.485+0000 7f3f31c77700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 0x7f3f18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:16.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.486+0000 7f3f31c77700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 0x7f3f18032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f3f1c0096a0 tx=0x7f3f1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.490+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f280273f0 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:14.742303+0000 mgr.smithi080.vyzlfg (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:16.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.694+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f2c0fad30 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.695+0000 7f3f22ffd700 1 -- 172.21.15.112:0/941215438 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f2801c890 con 0x7f3f2c0f51a0 2024-02-13T03:29:16.695 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:16.695 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:16.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.696+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 msgr2=0x7f3f18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.696+0000 7f3f3347a700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 0x7f3f18032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f3f1c0096a0 tx=0x7f3f1c006b40).stop 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 msgr2=0x7f3f2c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f3f28009830 tx=0x7f3f2800b040).stop 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 shutdown_connections 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3f1802ff60 0x7f3f18032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 --2- 172.21.15.112:0/941215438 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3f2c0f51a0 0x7f3f2c1054d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 >> 172.21.15.112:0/941215438 conn(0x7f3f2c0f09a0 msgr2=0x7f3f2c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 shutdown_connections 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:16.697+0000 7f3f3347a700 1 -- 172.21.15.112:0/941215438 wait complete. 2024-02-13T03:29:16.697 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:17.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:16.695359+0000 mon.smithi080 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.112:0/941215438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:18.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:18.274 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:18.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:16.742635+0000 mgr.smithi080.vyzlfg (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 -- 172.21.15.112:0/2470529414 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 msgr2=0x7f57080f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 --2- 172.21.15.112:0/2470529414 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f57080f6ea0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5704009fb0 tx=0x7f5704009b30).stop 2024-02-13T03:29:20.165 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 -- 172.21.15.112:0/2470529414 shutdown_connections 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 --2- 172.21.15.112:0/2470529414 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f57080f6ea0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 -- 172.21.15.112:0/2470529414 >> 172.21.15.112:0/2470529414 conn(0x7f57080f02e0 msgr2=0x7f57080f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 -- 172.21.15.112:0/2470529414 shutdown_connections 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.163+0000 7f570f461700 1 -- 172.21.15.112:0/2470529414 wait complete. 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570f461700 1 Processor -- start 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570f461700 1 -- start start 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570f461700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570f461700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57080f5370 con 0x7f57080f6a80 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570d1fd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570d1fd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:57098/0 (socket says 172.21.15.112:57098) 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.164+0000 7f570d1fd700 1 -- 172.21.15.112:0/744244794 learned_addr learned my addr 172.21.15.112:0/744244794 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f570d1fd700 1 -- 172.21.15.112:0/744244794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704005040 con 0x7f57080f6a80 2024-02-13T03:29:20.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f570d1fd700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f57040089f0 tx=0x7f570400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5704009320 con 0x7f57080f6a80 2024-02-13T03:29:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57080f5570 con 0x7f57080f6a80 2024-02-13T03:29:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5704013380 con 0x7f57080f6a80 2024-02-13T03:29:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5704014e20 con 0x7f57080f6a80 2024-02-13T03:29:20.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.165+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5708109b90 con 0x7f57080f6a80 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.166+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57040263a0 con 0x7f57080f6a80 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.166+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5708045ad0 con 0x7f57080f6a80 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.166+0000 7f56f9ffb700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 0x7f56f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.166+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5704041190 con 0x7f57080f6a80 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.166+0000 7f570c9fc700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 0x7f56f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:20.168 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.167+0000 7f570c9fc700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 0x7f56f40323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f56fc0096a0 tx=0x7f56fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:20.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.171+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5704012330 con 0x7f57080f6a80 2024-02-13T03:29:20.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.375+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57080fb030 con 0x7f57080f6a80 2024-02-13T03:29:20.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.376+0000 7f56f9ffb700 1 -- 172.21.15.112:0/744244794 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5704012330 con 0x7f57080f6a80 2024-02-13T03:29:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:20.376 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:20.377 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.377+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 msgr2=0x7f56f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.377+0000 7f570f461700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 0x7f56f40323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f56fc0096a0 tx=0x7f56fc006b40).stop 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.377+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 msgr2=0x7f5708109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.377+0000 7f570f461700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f57040089f0 tx=0x7f570400b040).stop 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 shutdown_connections 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f56f402ff10 0x7f56f40323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 --2- 172.21.15.112:0/744244794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57080f6a80 0x7f5708109460 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 >> 172.21.15.112:0/744244794 conn(0x7f57080f02e0 msgr2=0x7f57080f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 shutdown_connections 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:20.378+0000 7f570f461700 1 -- 172.21.15.112:0/744244794 wait complete. 2024-02-13T03:29:20.378 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:20.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:18.742956+0000 mgr.smithi080.vyzlfg (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:20.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:20.376108+0000 mon.smithi080 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.112:0/744244794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:21.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:21.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:22.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:20.743294+0000 mgr.smithi080.vyzlfg (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 -- 172.21.15.112:0/3841297167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 msgr2=0x7f8a8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/3841297167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8a80009fb0 tx=0x7f8a80009b30).stop 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 -- 172.21.15.112:0/3841297167 shutdown_connections 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/3841297167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 -- 172.21.15.112:0/3841297167 >> 172.21.15.112:0/3841297167 conn(0x7f8a8c0f0b70 msgr2=0x7f8a8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 -- 172.21.15.112:0/3841297167 shutdown_connections 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.816+0000 7f8a93dbc700 1 -- 172.21.15.112:0/3841297167 wait complete. 2024-02-13T03:29:23.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a93dbc700 1 Processor -- start 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a93dbc700 1 -- start start 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a93dbc700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a93dbc700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c060220 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a91b58700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a91b58700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42186/0 (socket says 172.21.15.112:42186) 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a91b58700 1 -- 172.21.15.112:0/2113706790 learned_addr learned my addr 172.21.15.112:0/2113706790 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.817+0000 7f8a91b58700 1 -- 172.21.15.112:0/2113706790 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80005040 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a91b58700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8a80005010 tx=0x7f8a8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a80009320 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c060420 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a80013380 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a80014e50 con 0x7f8a8c0f5410 2024-02-13T03:29:23.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.818+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c109bb0 con 0x7f8a8c0f5410 2024-02-13T03:29:23.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a800095f0 con 0x7f8a8c0f5410 2024-02-13T03:29:23.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c059100 con 0x7f8a8c0f5410 2024-02-13T03:29:23.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a7e7fc700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 0x7f8a780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:23.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a80040e70 con 0x7f8a8c0f5410 2024-02-13T03:29:23.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a91357700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 0x7f8a780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:23.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.819+0000 7f8a91357700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 0x7f8a780323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f8a880096a0 tx=0x7f8a88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:23.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:23.823+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a80014420 con 0x7f8a8c0f5410 2024-02-13T03:29:24.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.029+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a8c045ad0 con 0x7f8a8c0f5410 2024-02-13T03:29:24.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.030+0000 7f8a7e7fc700 1 -- 172.21.15.112:0/2113706790 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a80010230 con 0x7f8a8c0f5410 2024-02-13T03:29:24.030 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:24.030 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:24.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 msgr2=0x7f8a780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:24.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 0x7f8a780323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f8a880096a0 tx=0x7f8a88006b40).stop 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 msgr2=0x7f8a8c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8a80005010 tx=0x7f8a8000b040).stop 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 shutdown_connections 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a7802ff10 0x7f8a780323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 --2- 172.21.15.112:0/2113706790 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a8c0f5410 0x7f8a8c109500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 >> 172.21.15.112:0/2113706790 conn(0x7f8a8c0f0b70 msgr2=0x7f8a8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 shutdown_connections 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:24.031+0000 7f8a93dbc700 1 -- 172.21.15.112:0/2113706790 wait complete. 2024-02-13T03:29:24.032 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:24.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:22.743595+0000 mgr.smithi080.vyzlfg (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:24.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:24.030065+0000 mon.smithi080 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.112:0/2113706790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:25.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:25.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:26.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:24.743931+0000 mgr.smithi080.vyzlfg (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 -- 172.21.15.112:0/2891383093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 msgr2=0x7f9a780f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 --2- 172.21.15.112:0/2891383093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a780f90f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f9a74009fb0 tx=0x7f9a74009b30).stop 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 -- 172.21.15.112:0/2891383093 shutdown_connections 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 --2- 172.21.15.112:0/2891383093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a780f90f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 -- 172.21.15.112:0/2891383093 >> 172.21.15.112:0/2891383093 conn(0x7f9a780f0b70 msgr2=0x7f9a780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 -- 172.21.15.112:0/2891383093 shutdown_connections 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.669+0000 7f9a800f7700 1 -- 172.21.15.112:0/2891383093 wait complete. 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a800f7700 1 Processor -- start 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a800f7700 1 -- start start 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a800f7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a800f7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a78106c90 con 0x7f9a78060280 2024-02-13T03:29:27.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a7de93700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a7de93700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42202/0 (socket says 172.21.15.112:42202) 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.670+0000 7f9a7de93700 1 -- 172.21.15.112:0/3190841953 learned_addr learned my addr 172.21.15.112:0/3190841953 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a7de93700 1 -- 172.21.15.112:0/3190841953 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a74005040 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a7de93700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9a740089f0 tx=0x7f9a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a74009320 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a78106e90 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a74013380 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a74014e20 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.671+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a78107320 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a740095f0 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a78045ad0 con 0x7f9a78060280 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a6a7fc700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 0x7f9a640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:27.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a74040f50 con 0x7f9a78060280 2024-02-13T03:29:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a7d692700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 0x7f9a640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.672+0000 7f9a7d692700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 0x7f9a640323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9a6c0096a0 tx=0x7f9a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:27.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.676+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a74014420 con 0x7f9a78060280 2024-02-13T03:29:27.884 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.883+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a781081d0 con 0x7f9a78060280 2024-02-13T03:29:27.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.884+0000 7f9a6a7fc700 1 -- 172.21.15.112:0/3190841953 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a7401d980 con 0x7f9a78060280 2024-02-13T03:29:27.885 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:27.885 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 msgr2=0x7f9a640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 0x7f9a640323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9a6c0096a0 tx=0x7f9a6c006b40).stop 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 msgr2=0x7f9a781067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9a740089f0 tx=0x7f9a7400b040).stop 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 shutdown_connections 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a6402ff10 0x7f9a640323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 --2- 172.21.15.112:0/3190841953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a78060280 0x7f9a781067a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 >> 172.21.15.112:0/3190841953 conn(0x7f9a780f0b70 msgr2=0x7f9a780f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 shutdown_connections 2024-02-13T03:29:27.887 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:27.886+0000 7f9a800f7700 1 -- 172.21.15.112:0/3190841953 wait complete. 2024-02-13T03:29:27.888 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:28.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:26.744225+0000 mgr.smithi080.vyzlfg (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:28.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:27.885475+0000 mon.smithi080 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.112:0/3190841953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:29.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:29.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:30.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:28.744588+0000 mgr.smithi080.vyzlfg (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:31.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.325+0000 7fbd926dd700 1 -- 172.21.15.112:0/3691592427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 msgr2=0x7fbd8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:31.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.325+0000 7fbd926dd700 1 --2- 172.21.15.112:0/3691592427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fbd7c009fb0 tx=0x7fbd7c009b30).stop 2024-02-13T03:29:31.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.325+0000 7fbd926dd700 1 -- 172.21.15.112:0/3691592427 shutdown_connections 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.325+0000 7fbd926dd700 1 --2- 172.21.15.112:0/3691592427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.325+0000 7fbd926dd700 1 -- 172.21.15.112:0/3691592427 >> 172.21.15.112:0/3691592427 conn(0x7fbd8c0f0b70 msgr2=0x7fbd8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.326+0000 7fbd926dd700 1 -- 172.21.15.112:0/3691592427 shutdown_connections 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.326+0000 7fbd926dd700 1 -- 172.21.15.112:0/3691592427 wait complete. 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.326+0000 7fbd926dd700 1 Processor -- start 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.326+0000 7fbd926dd700 1 -- start start 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd926dd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd926dd700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c105bd0 con 0x7fbd8c0f5410 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd8bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd8bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33704/0 (socket says 172.21.15.112:33704) 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd8bfff700 1 -- 172.21.15.112:0/271430186 learned_addr learned my addr 172.21.15.112:0/271430186 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd8bfff700 1 -- 172.21.15.112:0/271430186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c005040 con 0x7fbd8c0f5410 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.327+0000 7fbd8bfff700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbd7c0089f0 tx=0x7fbd7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7c009320 con 0x7fbd8c0f5410 2024-02-13T03:29:31.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8c105dd0 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd7c013380 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7c01ed80 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8c106260 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd7c01e420 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.328+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8c045ad0 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.329+0000 7fbd88ff9700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.329+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd7c041150 con 0x7fbd8c0f5410 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.329+0000 7fbd8b7fe700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:31.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.329+0000 7fbd8b7fe700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbd800096a0 tx=0x7fbd80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:31.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.333+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd7c0095f0 con 0x7fbd8c0f5410 2024-02-13T03:29:31.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.539+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd8c107110 con 0x7fbd8c0f5410 2024-02-13T03:29:31.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.539+0000 7fbd88ff9700 1 -- 172.21.15.112:0/271430186 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd7c004490 con 0x7fbd8c0f5410 2024-02-13T03:29:31.540 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:31.540 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 msgr2=0x7fbd74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbd800096a0 tx=0x7fbd80006b40).stop 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 msgr2=0x7fbd8c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbd7c0089f0 tx=0x7fbd7c00b040).stop 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 shutdown_connections 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 --2- 172.21.15.112:0/271430186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbd8c0f5410 0x7fbd8c1056e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 >> 172.21.15.112:0/271430186 conn(0x7fbd8c0f0b70 msgr2=0x7fbd8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 shutdown_connections 2024-02-13T03:29:31.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:31.541+0000 7fbd926dd700 1 -- 172.21.15.112:0/271430186 wait complete. 2024-02-13T03:29:31.543 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:32.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:30.744908+0000 mgr.smithi080.vyzlfg (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:32.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:31.540796+0000 mon.smithi080 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.112:0/271430186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:33.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:33.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:34.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:32.745269+0000 mgr.smithi080.vyzlfg (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:35.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- 172.21.15.112:0/3076708165 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 msgr2=0x7fb8040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:35.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/3076708165 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb800009fb0 tx=0x7fb800009b30).stop 2024-02-13T03:29:35.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- 172.21.15.112:0/3076708165 shutdown_connections 2024-02-13T03:29:35.071 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/3076708165 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- 172.21.15.112:0/3076708165 >> 172.21.15.112:0/3076708165 conn(0x7fb8040f0b70 msgr2=0x7fb8040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- 172.21.15.112:0/3076708165 shutdown_connections 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- 172.21.15.112:0/3076708165 wait complete. 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 Processor -- start 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- start start 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.070+0000 7fb80c0d1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8040fb960 con 0x7fb8040f5410 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb809e6d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb809e6d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33724/0 (socket says 172.21.15.112:33724) 2024-02-13T03:29:35.072 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb809e6d700 1 -- 172.21.15.112:0/1677791445 learned_addr learned my addr 172.21.15.112:0/1677791445 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb809e6d700 1 -- 172.21.15.112:0/1677791445 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800005040 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb809e6d700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb800005010 tx=0x7fb80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800009320 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb800013380 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8040fbb00 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800014e50 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.071+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8040fbf90 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.072+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8000095f0 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.072+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb804059100 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.072+0000 7fb7f67fc700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 0x7fb7f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.072+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb800040fa0 con 0x7fb8040f5410 2024-02-13T03:29:35.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.072+0000 7fb80966c700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 0x7fb7f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:35.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.073+0000 7fb80966c700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 0x7fb7f00323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb7f80096a0 tx=0x7fb7f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:35.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.077+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb800014420 con 0x7fb8040f5410 2024-02-13T03:29:35.282 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.282+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8040fafa0 con 0x7fb8040f5410 2024-02-13T03:29:35.283 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.282+0000 7fb7f67fc700 1 -- 172.21.15.112:0/1677791445 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb800010230 con 0x7fb8040f5410 2024-02-13T03:29:35.283 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:35.283 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 msgr2=0x7fb7f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 0x7fb7f00323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb7f80096a0 tx=0x7fb7f8006b40).stop 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 msgr2=0x7fb8040fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb800005010 tx=0x7fb80000b040).stop 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 shutdown_connections 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb7f002ff10 0x7fb7f00323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 --2- 172.21.15.112:0/1677791445 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8040f5410 0x7fb8040fd4e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 >> 172.21.15.112:0/1677791445 conn(0x7fb8040f0b70 msgr2=0x7fb80405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 shutdown_connections 2024-02-13T03:29:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:35.284+0000 7fb80c0d1700 1 -- 172.21.15.112:0/1677791445 wait complete. 2024-02-13T03:29:35.286 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:35.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:35 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:35.283745+0000 mon.smithi080 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.112:0/1677791445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:36.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:34.745491+0000 mgr.smithi080.vyzlfg (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:36.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:36.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:38.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:36.745771+0000 mgr.smithi080.vyzlfg (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- 172.21.15.112:0/832296306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 msgr2=0x7f8d100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/832296306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d100f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8d00009fb0 tx=0x7f8d00009b30).stop 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- 172.21.15.112:0/832296306 shutdown_connections 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/832296306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d100f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- 172.21.15.112:0/832296306 >> 172.21.15.112:0/832296306 conn(0x7f8d100f09a0 msgr2=0x7f8d100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- 172.21.15.112:0/832296306 shutdown_connections 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- 172.21.15.112:0/832296306 wait complete. 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 Processor -- start 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.794+0000 7f8d16ccf700 1 -- start start 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d16ccf700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d16ccf700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d10106ac0 con 0x7f8d100f51a0 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d15ccd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d15ccd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33730/0 (socket says 172.21.15.112:33730) 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d15ccd700 1 -- 172.21.15.112:0/1934480644 learned_addr learned my addr 172.21.15.112:0/1934480644 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d15ccd700 1 -- 172.21.15.112:0/1934480644 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00005040 con 0x7f8d100f51a0 2024-02-13T03:29:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d15ccd700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8d00009830 tx=0x7f8d0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d00009320 con 0x7f8d100f51a0 2024-02-13T03:29:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d10106cc0 con 0x7f8d100f51a0 2024-02-13T03:29:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d00013380 con 0x7f8d100f51a0 2024-02-13T03:29:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d00014420 con 0x7f8d100f51a0 2024-02-13T03:29:38.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.795+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d10107150 con 0x7f8d100f51a0 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.796+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d0001e420 con 0x7f8d100f51a0 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.796+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d10045ad0 con 0x7f8d100f51a0 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.796+0000 7f8d067fc700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 0x7f8cfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.796+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d00040ee0 con 0x7f8d100f51a0 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.797+0000 7f8d154cc700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 0x7f8cfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:38.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.797+0000 7f8d154cc700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 0x7f8cfc032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f8d0c0096a0 tx=0x7f8d0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:38.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:38.801+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d0001c930 con 0x7f8d100f51a0 2024-02-13T03:29:39.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.006+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d100fad30 con 0x7f8d100f51a0 2024-02-13T03:29:39.007 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.006+0000 7f8d067fc700 1 -- 172.21.15.112:0/1934480644 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d0002c2d0 con 0x7f8d100f51a0 2024-02-13T03:29:39.007 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:39.007 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 msgr2=0x7f8cfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 0x7f8cfc032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f8d0c0096a0 tx=0x7f8d0c006b40).stop 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 msgr2=0x7f8d101065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8d00009830 tx=0x7f8d0000b040).stop 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 shutdown_connections 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8cfc02ff60 0x7f8cfc032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 --2- 172.21.15.112:0/1934480644 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d100f51a0 0x7f8d101065d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 >> 172.21.15.112:0/1934480644 conn(0x7f8d100f09a0 msgr2=0x7f8d1005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 shutdown_connections 2024-02-13T03:29:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:39.008+0000 7f8d16ccf700 1 -- 172.21.15.112:0/1934480644 wait complete. 2024-02-13T03:29:39.010 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:39.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:39.007733+0000 mon.smithi080 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.112:0/1934480644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:40.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:40.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:40.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:38.746068+0000 mgr.smithi080.vyzlfg (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.364+0000 7fb50ffff700 1 -- 172.21.15.112:0/781191722 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 msgr2=0x7fb5100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.364+0000 7fb50ffff700 1 --2- 172.21.15.112:0/781191722 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5100f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb500009fb0 tx=0x7fb500009b30).stop 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.364+0000 7fb50ffff700 1 -- 172.21.15.112:0/781191722 shutdown_connections 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.364+0000 7fb50ffff700 1 --2- 172.21.15.112:0/781191722 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5100f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.364+0000 7fb50ffff700 1 -- 172.21.15.112:0/781191722 >> 172.21.15.112:0/781191722 conn(0x7fb5100f09a0 msgr2=0x7fb5100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 -- 172.21.15.112:0/781191722 shutdown_connections 2024-02-13T03:29:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 -- 172.21.15.112:0/781191722 wait complete. 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 Processor -- start 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 -- start start 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.365+0000 7fb50ffff700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5101059c0 con 0x7fb5100f51a0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:55336/0 (socket says 172.21.15.112:55336) 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50effd700 1 -- 172.21.15.112:0/519368074 learned_addr learned my addr 172.21.15.112:0/519368074 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50effd700 1 -- 172.21.15.112:0/519368074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb500005040 con 0x7fb5100f51a0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50effd700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb5000089f0 tx=0x7fb50000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb500009320 con 0x7fb5100f51a0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb510105bc0 con 0x7fb5100f51a0 2024-02-13T03:29:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb500013380 con 0x7fb5100f51a0 2024-02-13T03:29:42.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb500014e20 con 0x7fb5100f51a0 2024-02-13T03:29:42.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.366+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb510106050 con 0x7fb5100f51a0 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.367+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb500014420 con 0x7fb5100f51a0 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.367+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb510045ad0 con 0x7fb5100f51a0 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.367+0000 7fb4f77fe700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 0x7fb4f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.367+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb500040ed0 con 0x7fb5100f51a0 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.367+0000 7fb50e7fc700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 0x7fb4f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.368+0000 7fb50e7fc700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 0x7fb4f80323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb5040096a0 tx=0x7fb504006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:42.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.372+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb500014790 con 0x7fb5100f51a0 2024-02-13T03:29:42.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.577+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb510106ea0 con 0x7fb5100f51a0 2024-02-13T03:29:42.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.578+0000 7fb4f77fe700 1 -- 172.21.15.112:0/519368074 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb50001d7f0 con 0x7fb5100f51a0 2024-02-13T03:29:42.579 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:42.579 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:42.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 msgr2=0x7fb4f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:42.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 0x7fb4f80323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb5040096a0 tx=0x7fb504006b40).stop 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 msgr2=0x7fb5101054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb5000089f0 tx=0x7fb50000b040).stop 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 shutdown_connections 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4f802ff10 0x7fb4f80323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 --2- 172.21.15.112:0/519368074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5100f51a0 0x7fb5101054d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 >> 172.21.15.112:0/519368074 conn(0x7fb5100f09a0 msgr2=0x7fb5100f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 shutdown_connections 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:42.580+0000 7fb50ffff700 1 -- 172.21.15.112:0/519368074 wait complete. 2024-02-13T03:29:42.581 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:42.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:40.746396+0000 mgr.smithi080.vyzlfg (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:43.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:42.579536+0000 mon.smithi080 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.112:0/519368074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:44.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:44.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:42.746755+0000 mgr.smithi080.vyzlfg (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- 172.21.15.112:0/50594178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 msgr2=0x7fe1780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 --2- 172.21.15.112:0/50594178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1780f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fe168009fb0 tx=0x7fe168009b30).stop 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- 172.21.15.112:0/50594178 shutdown_connections 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 --2- 172.21.15.112:0/50594178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1780f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- 172.21.15.112:0/50594178 >> 172.21.15.112:0/50594178 conn(0x7fe1780f09a0 msgr2=0x7fe1780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- 172.21.15.112:0/50594178 shutdown_connections 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- 172.21.15.112:0/50594178 wait complete. 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 Processor -- start 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.250+0000 7fe177fff700 1 -- start start 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe177fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe177fff700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1781059c0 con 0x7fe1780f51a0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe176ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe176ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:55350/0 (socket says 172.21.15.112:55350) 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe176ffd700 1 -- 172.21.15.112:0/3415375934 learned_addr learned my addr 172.21.15.112:0/3415375934 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe176ffd700 1 -- 172.21.15.112:0/3415375934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168005040 con 0x7fe1780f51a0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe176ffd700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe1680089f0 tx=0x7fe16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168009320 con 0x7fe1780f51a0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe178105bc0 con 0x7fe1780f51a0 2024-02-13T03:29:46.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe168013380 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe168014e20 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.251+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178106050 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.252+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe168014420 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.252+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178045ad0 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.252+0000 7fe15f7fe700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.252+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe168040fc0 con 0x7fe1780f51a0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.253+0000 7fe1767fc700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.253+0000 7fe1767fc700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 0x7fe1600323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:46.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.257+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe16806d2b0 con 0x7fe1780f51a0 2024-02-13T03:29:46.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.466+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe178106f00 con 0x7fe1780f51a0 2024-02-13T03:29:46.467 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.466+0000 7fe15f7fe700 1 -- 172.21.15.112:0/3415375934 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1680095f0 con 0x7fe1780f51a0 2024-02-13T03:29:46.467 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:46.467 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 msgr2=0x7fe1600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 0x7fe1600323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).stop 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 msgr2=0x7fe1781054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe1680089f0 tx=0x7fe16800b040).stop 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 shutdown_connections 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe16002ff10 0x7fe1600323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 --2- 172.21.15.112:0/3415375934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe1780f51a0 0x7fe1781054d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 >> 172.21.15.112:0/3415375934 conn(0x7fe1780f09a0 msgr2=0x7fe1780f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 shutdown_connections 2024-02-13T03:29:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:46.468+0000 7fe177fff700 1 -- 172.21.15.112:0/3415375934 wait complete. 2024-02-13T03:29:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:46.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:44.747144+0000 mgr.smithi080.vyzlfg (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:47.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:46.467694+0000 mon.smithi080 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.112:0/3415375934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:48.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:48.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:46.747484+0000 mgr.smithi080.vyzlfg (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- 172.21.15.112:0/1675772249 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 msgr2=0x7fa7f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/1675772249 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f80f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa7ec009fb0 tx=0x7fa7ec009b30).stop 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- 172.21.15.112:0/1675772249 shutdown_connections 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/1675772249 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f80f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- 172.21.15.112:0/1675772249 >> 172.21.15.112:0/1675772249 conn(0x7fa7f80f1c70 msgr2=0x7fa7f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- 172.21.15.112:0/1675772249 shutdown_connections 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- 172.21.15.112:0/1675772249 wait complete. 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 Processor -- start 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.946+0000 7fa7fdefb700 1 -- start start 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7fdefb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7fdefb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8069e20 con 0x7fa7f80f6510 2024-02-13T03:29:49.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:55370/0 (socket says 172.21.15.112:55370) 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7f77fe700 1 -- 172.21.15.112:0/758306192 learned_addr learned my addr 172.21.15.112:0/758306192 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7f77fe700 1 -- 172.21.15.112:0/758306192 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec005040 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7f77fe700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7ec005010 tx=0x7fa7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ec009320 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7ec013380 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f8069fc0 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7ec014e50 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.947+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f806a450 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.948+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7ec0095f0 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.948+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8059100 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.948+0000 7fa7dffff700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 0x7fa7e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.948+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7ec040fa0 con 0x7fa7f80f6510 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.949+0000 7fa7f6ffd700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 0x7fa7e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:49.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.949+0000 7fa7f6ffd700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 0x7fa7e00323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa7e80096a0 tx=0x7fa7e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:49.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:49.953+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7ec014420 con 0x7fa7f80f6510 2024-02-13T03:29:50.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.158+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7f80fc0a0 con 0x7fa7f80f6510 2024-02-13T03:29:50.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.159+0000 7fa7dffff700 1 -- 172.21.15.112:0/758306192 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7ec010230 con 0x7fa7f80f6510 2024-02-13T03:29:50.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:50.160 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 msgr2=0x7fa7e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 0x7fa7e00323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa7e80096a0 tx=0x7fa7e8006b40).stop 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 msgr2=0x7fa7f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa7ec005010 tx=0x7fa7ec00b040).stop 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 shutdown_connections 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa7e002ff10 0x7fa7e00323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:50.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 --2- 172.21.15.112:0/758306192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa7f80f6510 0x7fa7f806b9a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 >> 172.21.15.112:0/758306192 conn(0x7fa7f80f1c70 msgr2=0x7fa7f80f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 shutdown_connections 2024-02-13T03:29:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:50.160+0000 7fa7fdefb700 1 -- 172.21.15.112:0/758306192 wait complete. 2024-02-13T03:29:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:50.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:48.747833+0000 mgr.smithi080.vyzlfg (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:50.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:50.159988+0000 mon.smithi080 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.112:0/758306192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:51.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:51.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:50.748155+0000 mgr.smithi080.vyzlfg (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.673+0000 7f2211613700 1 -- 172.21.15.112:0/1624092736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 msgr2=0x7f220c0f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 --2- 172.21.15.112:0/1624092736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c0f6ea0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2200009fb0 tx=0x7f2200009b30).stop 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 -- 172.21.15.112:0/1624092736 shutdown_connections 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 --2- 172.21.15.112:0/1624092736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c0f6ea0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 -- 172.21.15.112:0/1624092736 >> 172.21.15.112:0/1624092736 conn(0x7f220c0f02e0 msgr2=0x7f220c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:53.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 -- 172.21.15.112:0/1624092736 shutdown_connections 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 -- 172.21.15.112:0/1624092736 wait complete. 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 Processor -- start 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.674+0000 7f2211613700 1 -- start start 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f2211613700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f2211613700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c105b90 con 0x7f220c0f6a80 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f220affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f220affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53384/0 (socket says 172.21.15.112:53384) 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f220affd700 1 -- 172.21.15.112:0/2800936554 learned_addr learned my addr 172.21.15.112:0/2800936554 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.675+0000 7f220affd700 1 -- 172.21.15.112:0/2800936554 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200005040 con 0x7f220c0f6a80 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f220affd700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f22000089f0 tx=0x7f220000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2200009320 con 0x7f220c0f6a80 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2200013380 con 0x7f220c0f6a80 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c105d90 con 0x7f220c0f6a80 2024-02-13T03:29:53.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2200014e20 con 0x7f220c0f6a80 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.676+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c106220 con 0x7f220c0f6a80 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.677+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2200014420 con 0x7f220c0f6a80 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.677+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c045ad0 con 0x7f220c0f6a80 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.677+0000 7f21f37fe700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 0x7f21f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.677+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2200040eb0 con 0x7f220c0f6a80 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.677+0000 7f220a7fc700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 0x7f21f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.678+0000 7f220a7fc700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 0x7f21f40323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f21fc0096a0 tx=0x7f21fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:53.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.682+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f220001dd50 con 0x7f220c0f6a80 2024-02-13T03:29:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.889+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f220c1070d0 con 0x7f220c0f6a80 2024-02-13T03:29:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.890+0000 7f21f37fe700 1 -- 172.21.15.112:0/2800936554 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f220001db70 con 0x7f220c0f6a80 2024-02-13T03:29:53.891 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:53.891 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 msgr2=0x7f21f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 0x7f21f40323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f21fc0096a0 tx=0x7f21fc006b40).stop 2024-02-13T03:29:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 msgr2=0x7f220c1056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f22000089f0 tx=0x7f220000b040).stop 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 shutdown_connections 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f21f402ff10 0x7f21f40323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 --2- 172.21.15.112:0/2800936554 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f220c0f6a80 0x7f220c1056a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 >> 172.21.15.112:0/2800936554 conn(0x7f220c0f02e0 msgr2=0x7f220c0f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 shutdown_connections 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:53.892+0000 7f2211613700 1 -- 172.21.15.112:0/2800936554 wait complete. 2024-02-13T03:29:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:52.748399+0000 mgr.smithi080.vyzlfg (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:53.891257+0000 mon.smithi080 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.112:0/2800936554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:55.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:55.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:29:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:54.748744+0000 mgr.smithi080.vyzlfg (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 -- 172.21.15.112:0/3474063033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 msgr2=0x7f9db00f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/3474063033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db00f4c00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9da0009fb0 tx=0x7f9da0009b30).stop 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 -- 172.21.15.112:0/3474063033 shutdown_connections 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/3474063033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db00f4c00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 -- 172.21.15.112:0/3474063033 >> 172.21.15.112:0/3474063033 conn(0x7f9db00f0260 msgr2=0x7f9db00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 -- 172.21.15.112:0/3474063033 shutdown_connections 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.448+0000 7f9db7a7b700 1 -- 172.21.15.112:0/3474063033 wait complete. 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db7a7b700 1 Processor -- start 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db7a7b700 1 -- start start 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db7a7b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:57.450 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db7a7b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db0106e90 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db5817700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db5817700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53396/0 (socket says 172.21.15.112:53396) 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.449+0000 7f9db5817700 1 -- 172.21.15.112:0/234082990 learned_addr learned my addr 172.21.15.112:0/234082990 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9db5817700 1 -- 172.21.15.112:0/234082990 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0005040 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9db5817700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9da00089f0 tx=0x7f9da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da0009320 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db0107090 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9da0013380 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da0014e20 con 0x7f9db00f47e0 2024-02-13T03:29:57.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.450+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db0107520 con 0x7f9db00f47e0 2024-02-13T03:29:57.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.451+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9da00095f0 con 0x7f9db00f47e0 2024-02-13T03:29:57.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.451+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db0045ad0 con 0x7f9db00f47e0 2024-02-13T03:29:57.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.451+0000 7f9da67fc700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 0x7f9d9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:29:57.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.451+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9da0040e90 con 0x7f9db00f47e0 2024-02-13T03:29:57.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.451+0000 7f9db5016700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 0x7f9d9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:29:57.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.452+0000 7f9db5016700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 0x7f9d9c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9dac0096a0 tx=0x7f9dac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:29:57.455 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.456+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9da0014420 con 0x7f9db00f47e0 2024-02-13T03:29:57.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.659+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9db005eb50 con 0x7f9db00f47e0 2024-02-13T03:29:57.660 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.660+0000 7f9da67fc700 1 -- 172.21.15.112:0/234082990 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9da001d980 con 0x7f9db00f47e0 2024-02-13T03:29:57.660 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:29:57.660 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:29:57.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 msgr2=0x7f9d9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 0x7f9d9c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9dac0096a0 tx=0x7f9dac006b40).stop 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 msgr2=0x7f9db01069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9da00089f0 tx=0x7f9da000b040).stop 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 shutdown_connections 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9d9c02ff10 0x7f9d9c0323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 --2- 172.21.15.112:0/234082990 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9db00f47e0 0x7f9db01069a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 >> 172.21.15.112:0/234082990 conn(0x7f9db00f0260 msgr2=0x7f9db00f0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 shutdown_connections 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:29:57.661+0000 7f9db7a7b700 1 -- 172.21.15.112:0/234082990 wait complete. 2024-02-13T03:29:57.662 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:29:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:56.855588+0000 mon.smithi080 (mon.0) 158 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:29:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:56.863893+0000 mon.smithi080 (mon.0) 159 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:29:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:56.749071+0000 mgr.smithi080.vyzlfg (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:29:58.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:29:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:29:57.660281+0000 mon.smithi080 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.112:0/234082990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:29:59.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:29:59.523 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:29:58.749358+0000 mgr.smithi080.vyzlfg (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.472+0000 7f8d2758b700 1 -- 172.21.15.112:0/3047526987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 msgr2=0x7f8d200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 --2- 172.21.15.112:0/3047526987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d200f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8d1c009fb0 tx=0x7f8d1c009b30).stop 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- 172.21.15.112:0/3047526987 shutdown_connections 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 --2- 172.21.15.112:0/3047526987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d200f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- 172.21.15.112:0/3047526987 >> 172.21.15.112:0/3047526987 conn(0x7f8d200f09a0 msgr2=0x7f8d200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- 172.21.15.112:0/3047526987 shutdown_connections 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- 172.21.15.112:0/3047526987 wait complete. 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 Processor -- start 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- start start 2024-02-13T03:30:01.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.473+0000 7f8d2758b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201059c0 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d26589700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d26589700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:60922/0 (socket says 172.21.15.112:60922) 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d26589700 1 -- 172.21.15.112:0/1906477036 learned_addr learned my addr 172.21.15.112:0/1906477036 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d26589700 1 -- 172.21.15.112:0/1906477036 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c005040 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d26589700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8d1c0089f0 tx=0x7f8d1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c009320 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d20105bc0 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d1c013380 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1c014e20 con 0x7f8d200f51a0 2024-02-13T03:30:01.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.474+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d20106050 con 0x7f8d200f51a0 2024-02-13T03:30:01.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.475+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d1c0095f0 con 0x7f8d200f51a0 2024-02-13T03:30:01.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.475+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d20045ad0 con 0x7f8d200f51a0 2024-02-13T03:30:01.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.475+0000 7f8d16ffd700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 0x7f8d0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:01.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.475+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d1c040f50 con 0x7f8d200f51a0 2024-02-13T03:30:01.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.475+0000 7f8d25d88700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 0x7f8d0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:01.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.476+0000 7f8d25d88700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 0x7f8d0c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8d100096a0 tx=0x7f8d10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:01.479 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.480+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d1c014420 con 0x7f8d200f51a0 2024-02-13T03:30:01.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.685+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d20106f00 con 0x7f8d200f51a0 2024-02-13T03:30:01.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.685+0000 7f8d16ffd700 1 -- 172.21.15.112:0/1906477036 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d1c01d980 con 0x7f8d200f51a0 2024-02-13T03:30:01.686 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:01.686 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 msgr2=0x7f8d0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 0x7f8d0c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8d100096a0 tx=0x7f8d10006b40).stop 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 msgr2=0x7f8d201054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8d1c0089f0 tx=0x7f8d1c00b040).stop 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 shutdown_connections 2024-02-13T03:30:01.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8d0c02ff10 0x7f8d0c0323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:01.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 --2- 172.21.15.112:0/1906477036 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8d200f51a0 0x7f8d201054d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:01.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 >> 172.21.15.112:0/1906477036 conn(0x7f8d200f09a0 msgr2=0x7f8d200f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:01.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 shutdown_connections 2024-02-13T03:30:01.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:01.687+0000 7f8d2758b700 1 -- 172.21.15.112:0/1906477036 wait complete. 2024-02-13T03:30:01.688 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:00.749697+0000 mgr.smithi080.vyzlfg (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:02.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:01.685819+0000 mon.smithi080 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.112:0/1906477036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:03.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:03.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:02.749979+0000 mgr.smithi080.vyzlfg (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:05.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 -- 172.21.15.112:0/3064255922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 msgr2=0x7f32d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 --2- 172.21.15.112:0/3064255922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d00f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f32c4009fb0 tx=0x7f32c4009b30).stop 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 -- 172.21.15.112:0/3064255922 shutdown_connections 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 --2- 172.21.15.112:0/3064255922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d00f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 -- 172.21.15.112:0/3064255922 >> 172.21.15.112:0/3064255922 conn(0x7f32d00f0b70 msgr2=0x7f32d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 -- 172.21.15.112:0/3064255922 shutdown_connections 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.152+0000 7f32d4900700 1 -- 172.21.15.112:0/3064255922 wait complete. 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32d4900700 1 Processor -- start 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32d4900700 1 -- start start 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32d4900700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:05.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32d4900700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d0106ce0 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32ce59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32ce59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:60952/0 (socket says 172.21.15.112:60952) 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32ce59c700 1 -- 172.21.15.112:0/2022094991 learned_addr learned my addr 172.21.15.112:0/2022094991 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.153+0000 7f32ce59c700 1 -- 172.21.15.112:0/2022094991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4005040 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32ce59c700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f32c4005010 tx=0x7f32c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32c4009320 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32c4013380 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0106ee0 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32c4014e50 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.154+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d0107370 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d0045ad0 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32c40095f0 con 0x7f32d00f5410 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32beffd700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 0x7f32b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32c4040e90 con 0x7f32d00f5410 2024-02-13T03:30:05.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32cdd9b700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 0x7f32b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:05.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.155+0000 7f32cdd9b700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 0x7f32b80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f32c00096a0 tx=0x7f32c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:05.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.159+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32c4014420 con 0x7f32d00f5410 2024-02-13T03:30:05.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.363+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32d0108220 con 0x7f32d00f5410 2024-02-13T03:30:05.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.364+0000 7f32beffd700 1 -- 172.21.15.112:0/2022094991 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32c4010230 con 0x7f32d00f5410 2024-02-13T03:30:05.365 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:05.365 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 msgr2=0x7f32b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 0x7f32b80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f32c00096a0 tx=0x7f32c0006b40).stop 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 msgr2=0x7f32d01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f32c4005010 tx=0x7f32c400b040).stop 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 shutdown_connections 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f32b802ff10 0x7f32b80323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 --2- 172.21.15.112:0/2022094991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f32d00f5410 0x7f32d01067f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 >> 172.21.15.112:0/2022094991 conn(0x7f32d00f0b70 msgr2=0x7f32d005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 shutdown_connections 2024-02-13T03:30:05.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:05.366+0000 7f32d4900700 1 -- 172.21.15.112:0/2022094991 wait complete. 2024-02-13T03:30:05.368 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:05.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:05.365333+0000 mon.smithi080 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.112:0/2022094991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:06.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:04.750235+0000 mgr.smithi080.vyzlfg (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:06.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:06.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 -- 172.21.15.112:0/3074792866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 msgr2=0x7fc1600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 --2- 172.21.15.112:0/3074792866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f90f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc148009fb0 tx=0x7fc148009b30).stop 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 -- 172.21.15.112:0/3074792866 shutdown_connections 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 --2- 172.21.15.112:0/3074792866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f90f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 -- 172.21.15.112:0/3074792866 >> 172.21.15.112:0/3074792866 conn(0x7fc1600f0b70 msgr2=0x7fc1600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 -- 172.21.15.112:0/3074792866 shutdown_connections 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.814+0000 7fc165a7e700 1 -- 172.21.15.112:0/3074792866 wait complete. 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc165a7e700 1 Processor -- start 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc165a7e700 1 -- start start 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc165a7e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:08.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc165a7e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1600f4ff0 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc15f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc15f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:60976/0 (socket says 172.21.15.112:60976) 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.815+0000 7fc15f7fe700 1 -- 172.21.15.112:0/1424399483 learned_addr learned my addr 172.21.15.112:0/1424399483 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc15f7fe700 1 -- 172.21.15.112:0/1424399483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148005040 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc15f7fe700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc148009830 tx=0x7fc14800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc148009320 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1600f5190 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc148013380 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc148014420 con 0x7fc160060280 2024-02-13T03:30:08.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.816+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1600f5620 con 0x7fc160060280 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc14801e420 con 0x7fc160060280 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc157fff700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 0x7fc14c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc160059100 con 0x7fc160060280 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc148040ef0 con 0x7fc160060280 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc15effd700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 0x7fc14c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:08.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.817+0000 7fc15effd700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 0x7fc14c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc1500096a0 tx=0x7fc150006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:08.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:08.822+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc14801cb10 con 0x7fc160060280 2024-02-13T03:30:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:06.750559+0000 mgr.smithi080.vyzlfg (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:09.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.026+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc160045ad0 con 0x7fc160060280 2024-02-13T03:30:09.027 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.027+0000 7fc157fff700 1 -- 172.21.15.112:0/1424399483 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc14801cb10 con 0x7fc160060280 2024-02-13T03:30:09.028 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:09.028 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:09.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 msgr2=0x7fc14c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:09.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 0x7fc14c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc1500096a0 tx=0x7fc150006b40).stop 2024-02-13T03:30:09.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 msgr2=0x7fc1600f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:09.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc148009830 tx=0x7fc14800b040).stop 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 shutdown_connections 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14c02ff60 0x7fc14c032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 --2- 172.21.15.112:0/1424399483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc160060280 0x7fc1600f6b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.028+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 >> 172.21.15.112:0/1424399483 conn(0x7fc1600f0b70 msgr2=0x7fc1600f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.029+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 shutdown_connections 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:09.029+0000 7fc165a7e700 1 -- 172.21.15.112:0/1424399483 wait complete. 2024-02-13T03:30:09.030 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:09.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:09.028314+0000 mon.smithi080 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.112:0/1424399483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:10.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:10.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:08.750808+0000 mgr.smithi080.vyzlfg (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 -- 172.21.15.112:0/1884895213 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 msgr2=0x7f8a340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 --2- 172.21.15.112:0/1884895213 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a340f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8a28009fb0 tx=0x7f8a28009b30).stop 2024-02-13T03:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 -- 172.21.15.112:0/1884895213 shutdown_connections 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 --2- 172.21.15.112:0/1884895213 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a340f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 -- 172.21.15.112:0/1884895213 >> 172.21.15.112:0/1884895213 conn(0x7f8a340f0b70 msgr2=0x7f8a340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 -- 172.21.15.112:0/1884895213 shutdown_connections 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.524+0000 7f8a3a040700 1 -- 172.21.15.112:0/1884895213 wait complete. 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a3a040700 1 Processor -- start 2024-02-13T03:30:12.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a3a040700 1 -- start start 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a3a040700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a3a040700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34106cc0 con 0x7f8a340f5410 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a337fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a337fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45448/0 (socket says 172.21.15.112:45448) 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.525+0000 7f8a337fe700 1 -- 172.21.15.112:0/2457293114 learned_addr learned my addr 172.21.15.112:0/2457293114 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a337fe700 1 -- 172.21.15.112:0/2457293114 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28005040 con 0x7f8a340f5410 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a337fe700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8a28005010 tx=0x7f8a2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a28009320 con 0x7f8a340f5410 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a34106ec0 con 0x7f8a340f5410 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a28013380 con 0x7f8a340f5410 2024-02-13T03:30:12.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a28014e50 con 0x7f8a340f5410 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.526+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a34107350 con 0x7f8a340f5410 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.527+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a280263a0 con 0x7f8a340f5410 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.527+0000 7f8a1bfff700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 0x7f8a1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.527+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a34045ad0 con 0x7f8a340f5410 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.527+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a28041160 con 0x7f8a340f5410 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.527+0000 7f8a32ffd700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 0x7f8a1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:12.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.528+0000 7f8a32ffd700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 0x7f8a1c0323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f8a240096a0 tx=0x7f8a24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:12.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.532+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a2801cb50 con 0x7f8a340f5410 2024-02-13T03:30:12.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.737+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a34108200 con 0x7f8a340f5410 2024-02-13T03:30:12.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.738+0000 7f8a1bfff700 1 -- 172.21.15.112:0/2457293114 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a2801cd60 con 0x7f8a340f5410 2024-02-13T03:30:12.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:12.740 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 msgr2=0x7f8a1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 0x7f8a1c0323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f8a240096a0 tx=0x7f8a24006b40).stop 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 msgr2=0x7f8a341067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8a28005010 tx=0x7f8a2800b040).stop 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 shutdown_connections 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a1c02ff10 0x7f8a1c0323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 --2- 172.21.15.112:0/2457293114 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a340f5410 0x7f8a341067d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:12.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 >> 172.21.15.112:0/2457293114 conn(0x7f8a340f0b70 msgr2=0x7f8a3405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 shutdown_connections 2024-02-13T03:30:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:12.740+0000 7f8a3a040700 1 -- 172.21.15.112:0/2457293114 wait complete. 2024-02-13T03:30:12.742 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:10.751102+0000 mgr.smithi080.vyzlfg (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:13.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:13 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:12.739546+0000 mon.smithi080 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.112:0/2457293114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:14.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:14.394 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:12.751444+0000 mgr.smithi080.vyzlfg (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 -- 172.21.15.112:0/1900995696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 msgr2=0x7f23640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 --2- 172.21.15.112:0/1900995696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23640f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2358009fb0 tx=0x7f2358009b30).stop 2024-02-13T03:30:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 -- 172.21.15.112:0/1900995696 shutdown_connections 2024-02-13T03:30:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 --2- 172.21.15.112:0/1900995696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23640f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 -- 172.21.15.112:0/1900995696 >> 172.21.15.112:0/1900995696 conn(0x7f23640f1c70 msgr2=0x7f23640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 -- 172.21.15.112:0/1900995696 shutdown_connections 2024-02-13T03:30:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.259+0000 7f23695f3700 1 -- 172.21.15.112:0/1900995696 wait complete. 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f23695f3700 1 Processor -- start 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f23695f3700 1 -- start start 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f23695f3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f23695f3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2364106cd0 con 0x7f23640f6510 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f2362ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f2362ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45460/0 (socket says 172.21.15.112:45460) 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.260+0000 7f2362ffd700 1 -- 172.21.15.112:0/3986246758 learned_addr learned my addr 172.21.15.112:0/3986246758 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f2362ffd700 1 -- 172.21.15.112:0/3986246758 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2358005040 con 0x7f23640f6510 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f2362ffd700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f23580089f0 tx=0x7f235800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2358009320 con 0x7f23640f6510 2024-02-13T03:30:16.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2364106ed0 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2358013380 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f235801ed80 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.261+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364107360 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.262+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f235801e420 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.262+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2364045ad0 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.262+0000 7f234b7fe700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 0x7f234c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.262+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2358041060 con 0x7f23640f6510 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.262+0000 7f23627fc700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 0x7f234c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:16.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.263+0000 7f23627fc700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 0x7f234c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:16.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.267+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f235801cb10 con 0x7f23640f6510 2024-02-13T03:30:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.470+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2364108210 con 0x7f23640f6510 2024-02-13T03:30:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.471+0000 7f234b7fe700 1 -- 172.21.15.112:0/3986246758 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f235802c2d0 con 0x7f23640f6510 2024-02-13T03:30:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:16.472 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 msgr2=0x7f234c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 0x7f234c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).stop 2024-02-13T03:30:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 msgr2=0x7f23641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f23580089f0 tx=0x7f235800b040).stop 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 shutdown_connections 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f234c02ff60 0x7f234c032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 --2- 172.21.15.112:0/3986246758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f23640f6510 0x7f23641067e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 >> 172.21.15.112:0/3986246758 conn(0x7f23640f1c70 msgr2=0x7f23640f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 shutdown_connections 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:16.472+0000 7f23695f3700 1 -- 172.21.15.112:0/3986246758 wait complete. 2024-02-13T03:30:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:14.751816+0000 mgr.smithi080.vyzlfg (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:16.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:16.472298+0000 mon.smithi080 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.112:0/3986246758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:18.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:18.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:18.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:16.752218+0000 mgr.smithi080.vyzlfg (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 -- 172.21.15.112:0/2767409318 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 msgr2=0x7feca80f8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 --2- 172.21.15.112:0/2767409318 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca80f8b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fec98009fb0 tx=0x7fec98009b30).stop 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 -- 172.21.15.112:0/2767409318 shutdown_connections 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 --2- 172.21.15.112:0/2767409318 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca80f8b20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 -- 172.21.15.112:0/2767409318 >> 172.21.15.112:0/2767409318 conn(0x7feca80f40e0 msgr2=0x7feca80f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:20.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 -- 172.21.15.112:0/2767409318 shutdown_connections 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 -- 172.21.15.112:0/2767409318 wait complete. 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.029+0000 7fecae266700 1 Processor -- start 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7fecae266700 1 -- start start 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7fecae266700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7fecae266700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8108ef0 con 0x7feca80f8700 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7feca7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7feca7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45484/0 (socket says 172.21.15.112:45484) 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7feca7fff700 1 -- 172.21.15.112:0/227872067 learned_addr learned my addr 172.21.15.112:0/227872067 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.030+0000 7feca7fff700 1 -- 172.21.15.112:0/227872067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98005040 con 0x7feca80f8700 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7feca7fff700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec98005010 tx=0x7fec9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec98009320 con 0x7feca80f8700 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca81090f0 con 0x7feca80f8700 2024-02-13T03:30:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec98013380 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec98014e50 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.031+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca8109580 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec980095f0 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca8045ad0 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7feca4ff9700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 0x7fec900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec98040fa0 con 0x7feca80f8700 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7feca77fe700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 0x7fec900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.032+0000 7feca77fe700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 0x7fec900323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fec9c0096a0 tx=0x7fec9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:20.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.037+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec98014420 con 0x7feca80f8700 2024-02-13T03:30:20.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.242+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feca810a430 con 0x7feca80f8700 2024-02-13T03:30:20.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.243+0000 7feca4ff9700 1 -- 172.21.15.112:0/227872067 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec98010230 con 0x7feca80f8700 2024-02-13T03:30:20.244 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:20.244 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:20.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 msgr2=0x7fec900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 0x7fec900323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fec9c0096a0 tx=0x7fec9c006b40).stop 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 msgr2=0x7feca8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec98005010 tx=0x7fec9800b040).stop 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 shutdown_connections 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fec9002ff10 0x7fec900323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 --2- 172.21.15.112:0/227872067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feca80f8700 0x7feca8108a00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 >> 172.21.15.112:0/227872067 conn(0x7feca80f40e0 msgr2=0x7feca80f5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 shutdown_connections 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:20.245+0000 7fecae266700 1 -- 172.21.15.112:0/227872067 wait complete. 2024-02-13T03:30:20.246 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:20.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:18.752584+0000 mgr.smithi080.vyzlfg (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:20.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:20.244260+0000 mon.smithi080 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.112:0/227872067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:21.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:21.971 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:20.752905+0000 mgr.smithi080.vyzlfg (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:23.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 -- 172.21.15.112:0/3750271447 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 msgr2=0x7fc9500f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:23.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3750271447 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f90f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc940009fb0 tx=0x7fc940009b30).stop 2024-02-13T03:30:23.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 -- 172.21.15.112:0/3750271447 shutdown_connections 2024-02-13T03:30:23.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3750271447 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f90f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 -- 172.21.15.112:0/3750271447 >> 172.21.15.112:0/3750271447 conn(0x7fc9500f0b70 msgr2=0x7fc9500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 -- 172.21.15.112:0/3750271447 shutdown_connections 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.907+0000 7fc9564c5700 1 -- 172.21.15.112:0/3750271447 wait complete. 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc9564c5700 1 Processor -- start 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc9564c5700 1 -- start start 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc9564c5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc9564c5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500f4ff0 con 0x7fc950060280 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc94ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc94ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:39040/0 (socket says 172.21.15.112:39040) 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.908+0000 7fc94ffff700 1 -- 172.21.15.112:0/3510186023 learned_addr learned my addr 172.21.15.112:0/3510186023 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc94ffff700 1 -- 172.21.15.112:0/3510186023 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc940005040 con 0x7fc950060280 2024-02-13T03:30:23.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc94ffff700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc9400089f0 tx=0x7fc94000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:23.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc940009320 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500f5190 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc940013380 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc940014e20 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.909+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9500f5620 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9400095f0 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc950059100 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc94cff9700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 0x7fc9380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc940040e90 con 0x7fc950060280 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc94f7fe700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 0x7fc9380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:23.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.910+0000 7fc94f7fe700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 0x7fc9380323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:23.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:23.915+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc940014420 con 0x7fc950060280 2024-02-13T03:30:24.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.119+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9500fc090 con 0x7fc950060280 2024-02-13T03:30:24.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.120+0000 7fc94cff9700 1 -- 172.21.15.112:0/3510186023 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc94001d980 con 0x7fc950060280 2024-02-13T03:30:24.121 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:24.121 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:24.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 msgr2=0x7fc9380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 0x7fc9380323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc9440096a0 tx=0x7fc944006b40).stop 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 msgr2=0x7fc9500f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc9400089f0 tx=0x7fc94000b040).stop 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 shutdown_connections 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc93802ff10 0x7fc9380323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 --2- 172.21.15.112:0/3510186023 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc950060280 0x7fc9500f6b70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 >> 172.21.15.112:0/3510186023 conn(0x7fc9500f0b70 msgr2=0x7fc9500f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 shutdown_connections 2024-02-13T03:30:24.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:24.122+0000 7fc9564c5700 1 -- 172.21.15.112:0/3510186023 wait complete. 2024-02-13T03:30:24.124 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:22.753167+0000 mgr.smithi080.vyzlfg (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:24.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:24.121498+0000 mon.smithi080 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.112:0/3510186023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:25.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:25.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:24.753513+0000 mgr.smithi080.vyzlfg (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 -- 172.21.15.112:0/2430326778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 msgr2=0x7fa3880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 --2- 172.21.15.112:0/2430326778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3880f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa37c009fb0 tx=0x7fa37c009b30).stop 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 -- 172.21.15.112:0/2430326778 shutdown_connections 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 --2- 172.21.15.112:0/2430326778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3880f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 -- 172.21.15.112:0/2430326778 >> 172.21.15.112:0/2430326778 conn(0x7fa3880f1c70 msgr2=0x7fa3880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 -- 172.21.15.112:0/2430326778 shutdown_connections 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.796+0000 7fa38db8d700 1 -- 172.21.15.112:0/2430326778 wait complete. 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.797+0000 7fa38db8d700 1 Processor -- start 2024-02-13T03:30:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.797+0000 7fa38db8d700 1 -- start start 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.797+0000 7fa38db8d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.797+0000 7fa38db8d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa388106cd0 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa3877fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa3877fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:39058/0 (socket says 172.21.15.112:39058) 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa3877fe700 1 -- 172.21.15.112:0/3904658816 learned_addr learned my addr 172.21.15.112:0/3904658816 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa3877fe700 1 -- 172.21.15.112:0/3904658816 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c005040 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa3877fe700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa37c005010 tx=0x7fa37c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c009320 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa388106ed0 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa37c013380 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c014e50 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.798+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa388107360 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.799+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa37c0095f0 con 0x7fa3880f6510 2024-02-13T03:30:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.799+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa388045ad0 con 0x7fa3880f6510 2024-02-13T03:30:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.799+0000 7fa36ffff700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 0x7fa3700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.799+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa37c040e90 con 0x7fa3880f6510 2024-02-13T03:30:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.799+0000 7fa386ffd700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 0x7fa3700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.800+0000 7fa386ffd700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 0x7fa3700323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa3780096a0 tx=0x7fa378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:27.804+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa37c014420 con 0x7fa3880f6510 2024-02-13T03:30:28.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.009+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa388108210 con 0x7fa3880f6510 2024-02-13T03:30:28.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.010+0000 7fa36ffff700 1 -- 172.21.15.112:0/3904658816 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa37c010230 con 0x7fa3880f6510 2024-02-13T03:30:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:28.011 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:28.012 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 msgr2=0x7fa3700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 0x7fa3700323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa3780096a0 tx=0x7fa378006b40).stop 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 msgr2=0x7fa3881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa37c005010 tx=0x7fa37c00b040).stop 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 shutdown_connections 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa37002ff10 0x7fa3700323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 --2- 172.21.15.112:0/3904658816 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa3880f6510 0x7fa3881067e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 >> 172.21.15.112:0/3904658816 conn(0x7fa3880f1c70 msgr2=0x7fa3880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.011+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 shutdown_connections 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:28.012+0000 7fa38db8d700 1 -- 172.21.15.112:0/3904658816 wait complete. 2024-02-13T03:30:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:26.753858+0000 mgr.smithi080.vyzlfg (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:28.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:28.011236+0000 mon.smithi080 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.112:0/3904658816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:29.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:29.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:28.754253+0000 mgr.smithi080.vyzlfg (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:31.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 -- 172.21.15.112:0/3919824481 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 msgr2=0x7fc4140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:31.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 --2- 172.21.15.112:0/3919824481 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4140f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fc410009fb0 tx=0x7fc410009b30).stop 2024-02-13T03:30:31.558 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 -- 172.21.15.112:0/3919824481 shutdown_connections 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 --2- 172.21.15.112:0/3919824481 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4140f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 -- 172.21.15.112:0/3919824481 >> 172.21.15.112:0/3919824481 conn(0x7fc4140f1c70 msgr2=0x7fc4140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 -- 172.21.15.112:0/3919824481 shutdown_connections 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.525+0000 7fc41c208700 1 -- 172.21.15.112:0/3919824481 wait complete. 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc41c208700 1 Processor -- start 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc41c208700 1 -- start start 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc41c208700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc41c208700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414106cd0 con 0x7fc4140f6510 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc419fa4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc419fa4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46014/0 (socket says 172.21.15.112:46014) 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc419fa4700 1 -- 172.21.15.112:0/2477314250 learned_addr learned my addr 172.21.15.112:0/2477314250 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:31.559 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc419fa4700 1 -- 172.21.15.112:0/2477314250 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410005040 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.526+0000 7fc419fa4700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc410005010 tx=0x7fc41000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc410009320 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc410013380 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc414106ed0 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc410014e50 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc414107360 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.527+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc414045ad0 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.528+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4100095f0 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.528+0000 7fc40a7fc700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 0x7fc4000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.528+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc410040e60 con 0x7fc4140f6510 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.528+0000 7fc4197a3700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 0x7fc4000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:31.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.528+0000 7fc4197a3700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 0x7fc4000323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:31.561 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.532+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc410014420 con 0x7fc4140f6510 2024-02-13T03:30:31.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.739+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc414108210 con 0x7fc4140f6510 2024-02-13T03:30:31.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.739+0000 7fc40a7fc700 1 -- 172.21.15.112:0/2477314250 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc41001d980 con 0x7fc4140f6510 2024-02-13T03:30:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:31.741 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 msgr2=0x7fc4000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 0x7fc4000323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).stop 2024-02-13T03:30:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 msgr2=0x7fc4141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc410005010 tx=0x7fc41000b040).stop 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 shutdown_connections 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc40002ff10 0x7fc4000323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 --2- 172.21.15.112:0/2477314250 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4140f6510 0x7fc4141067e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 >> 172.21.15.112:0/2477314250 conn(0x7fc4140f1c70 msgr2=0x7fc4140f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 shutdown_connections 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:31.741+0000 7fc41c208700 1 -- 172.21.15.112:0/2477314250 wait complete. 2024-02-13T03:30:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:30.754542+0000 mgr.smithi080.vyzlfg (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:32.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:31.740987+0000 mon.smithi080 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.112:0/2477314250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:33.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:33.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:34.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:32.754867+0000 mgr.smithi080.vyzlfg (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:35.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.366+0000 7f7eb9e72700 1 -- 172.21.15.112:0/661724045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 msgr2=0x7f7eb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:35.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.366+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/661724045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb40f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7ea4009fb0 tx=0x7f7ea4009b30).stop 2024-02-13T03:30:35.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.366+0000 7f7eb9e72700 1 -- 172.21.15.112:0/661724045 shutdown_connections 2024-02-13T03:30:35.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/661724045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb40f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:35.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 -- 172.21.15.112:0/661724045 >> 172.21.15.112:0/661724045 conn(0x7f7eb40f1c70 msgr2=0x7f7eb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:35.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 -- 172.21.15.112:0/661724045 shutdown_connections 2024-02-13T03:30:35.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 -- 172.21.15.112:0/661724045 wait complete. 2024-02-13T03:30:35.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 Processor -- start 2024-02-13T03:30:35.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 -- start start 2024-02-13T03:30:35.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:35.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.367+0000 7f7eb9e72700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4106cd0 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46032/0 (socket says 172.21.15.112:46032) 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb37fe700 1 -- 172.21.15.112:0/1264762240 learned_addr learned my addr 172.21.15.112:0/1264762240 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb37fe700 1 -- 172.21.15.112:0/1264762240 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea4005040 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb37fe700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7ea4005010 tx=0x7f7ea400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea4009320 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ea4013380 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb4106ed0 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea4014e50 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.368+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb4107360 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.369+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ea40095f0 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.369+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb4045ad0 con 0x7f7eb40f6510 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.369+0000 7f7e9bfff700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 0x7f7e9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:35.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.369+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ea4040e70 con 0x7f7eb40f6510 2024-02-13T03:30:35.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.369+0000 7f7eb2ffd700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 0x7f7e9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:35.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.370+0000 7f7eb2ffd700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 0x7f7e9c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7ea80096a0 tx=0x7f7ea8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:35.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.374+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ea4014420 con 0x7f7eb40f6510 2024-02-13T03:30:35.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.579+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7eb4108210 con 0x7f7eb40f6510 2024-02-13T03:30:35.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.579+0000 7f7e9bfff700 1 -- 172.21.15.112:0/1264762240 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ea4010230 con 0x7f7eb40f6510 2024-02-13T03:30:35.579 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 msgr2=0x7f7e9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 0x7f7e9c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7ea80096a0 tx=0x7f7ea8006b40).stop 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 msgr2=0x7f7eb41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7ea4005010 tx=0x7f7ea400b040).stop 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 shutdown_connections 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7e9c02ff10 0x7f7e9c0323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 --2- 172.21.15.112:0/1264762240 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7eb40f6510 0x7f7eb41067e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 >> 172.21.15.112:0/1264762240 conn(0x7f7eb40f1c70 msgr2=0x7f7eb40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 shutdown_connections 2024-02-13T03:30:35.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:35.581+0000 7f7eb9e72700 1 -- 172.21.15.112:0/1264762240 wait complete. 2024-02-13T03:30:35.582 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:34.755213+0000 mgr.smithi080.vyzlfg (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:36.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:35.579612+0000 mon.smithi080 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.112:0/1264762240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:37.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:37.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:36.755542+0000 mgr.smithi080.vyzlfg (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:39.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 -- 172.21.15.112:0/2672650821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 msgr2=0x7f5e7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 --2- 172.21.15.112:0/2672650821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c0f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5e6c009fb0 tx=0x7f5e6c009b30).stop 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 -- 172.21.15.112:0/2672650821 shutdown_connections 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 --2- 172.21.15.112:0/2672650821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c0f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 -- 172.21.15.112:0/2672650821 >> 172.21.15.112:0/2672650821 conn(0x7f5e7c0f0b70 msgr2=0x7f5e7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 -- 172.21.15.112:0/2672650821 shutdown_connections 2024-02-13T03:30:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.098+0000 7f5e8084b700 1 -- 172.21.15.112:0/2672650821 wait complete. 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.099+0000 7f5e8084b700 1 Processor -- start 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.099+0000 7f5e8084b700 1 -- start start 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.099+0000 7f5e8084b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.099+0000 7f5e8084b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7c105bd0 con 0x7f5e7c0f5410 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e7a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e7a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46052/0 (socket says 172.21.15.112:46052) 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e7a59c700 1 -- 172.21.15.112:0/3493612143 learned_addr learned my addr 172.21.15.112:0/3493612143 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e7a59c700 1 -- 172.21.15.112:0/3493612143 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c005040 con 0x7f5e7c0f5410 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e7a59c700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5e6c005010 tx=0x7f5e6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:39.101 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6c009320 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7c105dd0 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e6c013380 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6c014e50 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.100+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7c106260 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.101+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e6c0095f0 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.101+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7c045ad0 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.101+0000 7f5e72ffd700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 0x7f5e680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.101+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e6c040e90 con 0x7f5e7c0f5410 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.101+0000 7f5e79d9b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 0x7f5e680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:39.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.102+0000 7f5e79d9b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 0x7f5e680323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5e640096a0 tx=0x7f5e64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:39.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.106+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e6c014420 con 0x7f5e7c0f5410 2024-02-13T03:30:39.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.313+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e7c107110 con 0x7f5e7c0f5410 2024-02-13T03:30:39.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.313+0000 7f5e72ffd700 1 -- 172.21.15.112:0/3493612143 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e6c010230 con 0x7f5e7c0f5410 2024-02-13T03:30:39.313 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:39.313 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:39.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 msgr2=0x7f5e680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 0x7f5e680323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5e640096a0 tx=0x7f5e64006b40).stop 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 msgr2=0x7f5e7c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5e6c005010 tx=0x7f5e6c00b040).stop 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 shutdown_connections 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e6802ff10 0x7f5e680323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 --2- 172.21.15.112:0/3493612143 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e7c0f5410 0x7f5e7c1056e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 >> 172.21.15.112:0/3493612143 conn(0x7f5e7c0f0b70 msgr2=0x7f5e7c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 shutdown_connections 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:39.315+0000 7f5e8084b700 1 -- 172.21.15.112:0/3493612143 wait complete. 2024-02-13T03:30:39.315 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:39.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:39.313561+0000 mon.smithi080 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.112:0/3493612143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:38.755939+0000 mgr.smithi080.vyzlfg (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:40.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:40.986 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:42.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.913+0000 7f2e95616700 1 -- 172.21.15.112:0/2471039162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 msgr2=0x7f2e900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.913+0000 7f2e95616700 1 --2- 172.21.15.112:0/2471039162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e900f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2e80009fb0 tx=0x7f2e80009b30).stop 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- 172.21.15.112:0/2471039162 shutdown_connections 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 --2- 172.21.15.112:0/2471039162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e900f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- 172.21.15.112:0/2471039162 >> 172.21.15.112:0/2471039162 conn(0x7f2e900f1c70 msgr2=0x7f2e900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- 172.21.15.112:0/2471039162 shutdown_connections 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- 172.21.15.112:0/2471039162 wait complete. 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 Processor -- start 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- start start 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e95616700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90104a80 con 0x7f2e900f6510 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e8effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:42.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e8effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45762/0 (socket says 172.21.15.112:45762) 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.914+0000 7f2e8effd700 1 -- 172.21.15.112:0/2462414320 learned_addr learned my addr 172.21.15.112:0/2462414320 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e8effd700 1 -- 172.21.15.112:0/2462414320 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80005040 con 0x7f2e900f6510 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e8effd700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2e800089f0 tx=0x7f2e8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80009320 con 0x7f2e900f6510 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e90104c80 con 0x7f2e900f6510 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e80013380 con 0x7f2e900f6510 2024-02-13T03:30:42.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e8001ed80 con 0x7f2e900f6510 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.915+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e90105110 con 0x7f2e900f6510 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.916+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e8001e420 con 0x7f2e900f6510 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.916+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90045ad0 con 0x7f2e900f6510 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.916+0000 7f2e877fe700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 0x7f2e7c036870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.916+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e80040f30 con 0x7f2e900f6510 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.916+0000 7f2e87fff700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 0x7f2e7c036870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:42.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.917+0000 7f2e87fff700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 0x7f2e7c036870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2e780096a0 tx=0x7f2e78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:42.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:42.921+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e8006d2b0 con 0x7f2e900f6510 2024-02-13T03:30:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:40.756329+0000 mgr.smithi080.vyzlfg (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:43.124 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.125+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e9006b3c0 con 0x7f2e900f6510 2024-02-13T03:30:43.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.125+0000 7f2e877fe700 1 -- 172.21.15.112:0/2462414320 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e800095f0 con 0x7f2e900f6510 2024-02-13T03:30:43.125 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:43.125 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 msgr2=0x7f2e7c036870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 0x7f2e7c036870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2e780096a0 tx=0x7f2e78006b40).stop 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 msgr2=0x7f2e90104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2e800089f0 tx=0x7f2e8000b040).stop 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 shutdown_connections 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2e7c0343b0 0x7f2e7c036870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 --2- 172.21.15.112:0/2462414320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e900f6510 0x7f2e90104590 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 >> 172.21.15.112:0/2462414320 conn(0x7f2e900f1c70 msgr2=0x7f2e900f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 shutdown_connections 2024-02-13T03:30:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:43.127+0000 7f2e95616700 1 -- 172.21.15.112:0/2462414320 wait complete. 2024-02-13T03:30:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:43.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:43.125588+0000 mon.smithi080 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.112:0/2462414320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:44.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:44.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:42.756678+0000 mgr.smithi080.vyzlfg (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 -- 172.21.15.112:0/1390438829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 msgr2=0x7fa1940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 --2- 172.21.15.112:0/1390438829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1940f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa184009fb0 tx=0x7fa184009b30).stop 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 -- 172.21.15.112:0/1390438829 shutdown_connections 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 --2- 172.21.15.112:0/1390438829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1940f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 -- 172.21.15.112:0/1390438829 >> 172.21.15.112:0/1390438829 conn(0x7fa1940f0b70 msgr2=0x7fa1940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 -- 172.21.15.112:0/1390438829 shutdown_connections 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.640+0000 7fa199e39700 1 -- 172.21.15.112:0/1390438829 wait complete. 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.641+0000 7fa199e39700 1 Processor -- start 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.641+0000 7fa199e39700 1 -- start start 2024-02-13T03:30:46.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.641+0000 7fa199e39700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.641+0000 7fa199e39700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194106cd0 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa1937fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa1937fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45788/0 (socket says 172.21.15.112:45788) 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa1937fe700 1 -- 172.21.15.112:0/240977182 learned_addr learned my addr 172.21.15.112:0/240977182 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa1937fe700 1 -- 172.21.15.112:0/240977182 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa184005040 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa1937fe700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa184005010 tx=0x7fa18400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa184009320 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa184013380 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa194106ed0 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa184014e50 con 0x7fa1940f5410 2024-02-13T03:30:46.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.642+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa194107360 con 0x7fa1940f5410 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.643+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1840263a0 con 0x7fa1940f5410 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.643+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa194045ad0 con 0x7fa1940f5410 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.643+0000 7fa18bfff700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 0x7fa1800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.643+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa184040fc0 con 0x7fa1940f5410 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.643+0000 7fa192ffd700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 0x7fa1800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:46.645 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.644+0000 7fa192ffd700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 0x7fa1800323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa17c0096a0 tx=0x7fa17c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:46.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.648+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa184012510 con 0x7fa1940f5410 2024-02-13T03:30:46.853 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.853+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa194108210 con 0x7fa1940f5410 2024-02-13T03:30:46.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.853+0000 7fa18bfff700 1 -- 172.21.15.112:0/240977182 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa184012330 con 0x7fa1940f5410 2024-02-13T03:30:46.855 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:46.855 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 msgr2=0x7fa1800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 0x7fa1800323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa17c0096a0 tx=0x7fa17c006b40).stop 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 msgr2=0x7fa1941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa184005010 tx=0x7fa18400b040).stop 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 shutdown_connections 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa18002ff10 0x7fa1800323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:46.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 --2- 172.21.15.112:0/240977182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1940f5410 0x7fa1941067e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:46.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 >> 172.21.15.112:0/240977182 conn(0x7fa1940f0b70 msgr2=0x7fa1940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:46.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 shutdown_connections 2024-02-13T03:30:46.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:46.855+0000 7fa199e39700 1 -- 172.21.15.112:0/240977182 wait complete. 2024-02-13T03:30:46.857 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:44.756991+0000 mgr.smithi080.vyzlfg (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:47.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:46.854719+0000 mon.smithi080 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.112:0/240977182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:48.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:48.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:46.757328+0000 mgr.smithi080.vyzlfg (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 -- 172.21.15.112:0/2729240601 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 msgr2=0x7f69540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 --2- 172.21.15.112:0/2729240601 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69540f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6948009fb0 tx=0x7f6948009b30).stop 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 -- 172.21.15.112:0/2729240601 shutdown_connections 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 --2- 172.21.15.112:0/2729240601 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69540f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 -- 172.21.15.112:0/2729240601 >> 172.21.15.112:0/2729240601 conn(0x7f69540f1c70 msgr2=0x7f69540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 -- 172.21.15.112:0/2729240601 shutdown_connections 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.407+0000 7f6959b00700 1 -- 172.21.15.112:0/2729240601 wait complete. 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f6959b00700 1 Processor -- start 2024-02-13T03:30:50.411 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f6959b00700 1 -- start start 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f6959b00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f6959b00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6954106cd0 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f69537fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f69537fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45800/0 (socket says 172.21.15.112:45800) 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f69537fe700 1 -- 172.21.15.112:0/1052062145 learned_addr learned my addr 172.21.15.112:0/1052062145 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.408+0000 7f69537fe700 1 -- 172.21.15.112:0/1052062145 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948005040 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f69537fe700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6948005010 tx=0x7f694800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6948009320 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6954106ed0 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6948013380 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6948014e50 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.409+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6954107360 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6948014420 con 0x7f69540f6510 2024-02-13T03:30:50.412 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f693bfff700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 0x7f693c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:50.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6954045ad0 con 0x7f69540f6510 2024-02-13T03:30:50.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6948040e30 con 0x7f69540f6510 2024-02-13T03:30:50.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f6952ffd700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 0x7f693c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:50.413 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.410+0000 7f6952ffd700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 0x7f693c0323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f69440096a0 tx=0x7f6944006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:50.415 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.415+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69480095f0 con 0x7f69540f6510 2024-02-13T03:30:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.621+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6954108210 con 0x7f69540f6510 2024-02-13T03:30:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.621+0000 7f693bfff700 1 -- 172.21.15.112:0/1052062145 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f694801db70 con 0x7f69540f6510 2024-02-13T03:30:50.622 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:50.622 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 msgr2=0x7f693c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 0x7f693c0323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f69440096a0 tx=0x7f6944006b40).stop 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 msgr2=0x7f69541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6948005010 tx=0x7f694800b040).stop 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 shutdown_connections 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f693c02ff10 0x7f693c0323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 --2- 172.21.15.112:0/1052062145 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69540f6510 0x7f69541067e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:50.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 >> 172.21.15.112:0/1052062145 conn(0x7f69540f1c70 msgr2=0x7f69540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:50.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 shutdown_connections 2024-02-13T03:30:50.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:50.623+0000 7f6959b00700 1 -- 172.21.15.112:0/1052062145 wait complete. 2024-02-13T03:30:50.625 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:48.757636+0000 mgr.smithi080.vyzlfg (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:51.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:50.622899+0000 mon.smithi080 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.112:0/1052062145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:52.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:52.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:50.757983+0000 mgr.smithi080.vyzlfg (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.182+0000 7f287b0ef700 1 -- 172.21.15.112:0/2159581194 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 msgr2=0x7f28740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.182+0000 7f287b0ef700 1 --2- 172.21.15.112:0/2159581194 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f55c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2864009fb0 tx=0x7f2864009b30).stop 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- 172.21.15.112:0/2159581194 shutdown_connections 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 --2- 172.21.15.112:0/2159581194 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f55c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- 172.21.15.112:0/2159581194 >> 172.21.15.112:0/2159581194 conn(0x7f28740f09a0 msgr2=0x7f28740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- 172.21.15.112:0/2159581194 shutdown_connections 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- 172.21.15.112:0/2159581194 wait complete. 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 Processor -- start 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- start start 2024-02-13T03:30:54.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287b0ef700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28740f13a0 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287a0ed700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287a0ed700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53354/0 (socket says 172.21.15.112:53354) 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.183+0000 7f287a0ed700 1 -- 172.21.15.112:0/687688802 learned_addr learned my addr 172.21.15.112:0/687688802 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f287a0ed700 1 -- 172.21.15.112:0/687688802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864005040 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f287a0ed700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f28640089f0 tx=0x7f286400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2864009320 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28740f1540 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2864013380 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2864014e20 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.184+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28740f19d0 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.185+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28640095f0 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.185+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2874059100 con 0x7f28740f51a0 2024-02-13T03:30:54.187 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.185+0000 7f286affd700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 0x7f28600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:54.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.185+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2864040f50 con 0x7f28740f51a0 2024-02-13T03:30:54.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.185+0000 7f28798ec700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 0x7f28600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:54.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.186+0000 7f28798ec700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 0x7f28600323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f28700096a0 tx=0x7f2870006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.190+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2864014420 con 0x7f28740f51a0 2024-02-13T03:30:54.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.394+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2874045ad0 con 0x7f28740f51a0 2024-02-13T03:30:54.395 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.394+0000 7f286affd700 1 -- 172.21.15.112:0/687688802 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f286401d980 con 0x7f28740f51a0 2024-02-13T03:30:54.395 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:54.395 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 msgr2=0x7f28600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 0x7f28600323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f28700096a0 tx=0x7f2870006b40).stop 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 msgr2=0x7f28740f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f28640089f0 tx=0x7f286400b040).stop 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 shutdown_connections 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f286002ff10 0x7f28600323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 --2- 172.21.15.112:0/687688802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f28740f51a0 0x7f28740f2f20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 >> 172.21.15.112:0/687688802 conn(0x7f28740f09a0 msgr2=0x7f287405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 shutdown_connections 2024-02-13T03:30:54.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:54.396+0000 7f287b0ef700 1 -- 172.21.15.112:0/687688802 wait complete. 2024-02-13T03:30:54.398 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:52.758288+0000 mgr.smithi080.vyzlfg (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:54.395685+0000 mon.smithi080 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.112:0/687688802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:56.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:56.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:30:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:54.758613+0000 mgr.smithi080.vyzlfg (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:56.856591+0000 mon.smithi080 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:30:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:56.864927+0000 mon.smithi080 (mon.0) 177 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- 172.21.15.112:0/3292020475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 msgr2=0x7fcaac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 --2- 172.21.15.112:0/3292020475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac0f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fcaa8009fb0 tx=0x7fcaa8009b30).stop 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- 172.21.15.112:0/3292020475 shutdown_connections 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 --2- 172.21.15.112:0/3292020475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac0f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- 172.21.15.112:0/3292020475 >> 172.21.15.112:0/3292020475 conn(0x7fcaac0f09a0 msgr2=0x7fcaac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- 172.21.15.112:0/3292020475 shutdown_connections 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- 172.21.15.112:0/3292020475 wait complete. 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 Processor -- start 2024-02-13T03:30:58.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.028+0000 7fcab2efc700 1 -- start start 2024-02-13T03:30:58.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab2efc700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:58.030 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab2efc700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac106ac0 con 0x7fcaac0f51a0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab1efa700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab1efa700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53376/0 (socket says 172.21.15.112:53376) 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab1efa700 1 -- 172.21.15.112:0/2949274126 learned_addr learned my addr 172.21.15.112:0/2949274126 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab1efa700 1 -- 172.21.15.112:0/2949274126 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8005040 con 0x7fcaac0f51a0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab1efa700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fcaa80089f0 tx=0x7fcaa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa8009320 con 0x7fcaac0f51a0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac106cc0 con 0x7fcaac0f51a0 2024-02-13T03:30:58.031 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaa8013380 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa8014e20 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.029+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac107150 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.030+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaa80095f0 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.030+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac045ad0 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.030+0000 7fcaa27fc700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 0x7fca98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.030+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaa8040e80 con 0x7fcaac0f51a0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.030+0000 7fcab16f9700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 0x7fca98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:30:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.031+0000 7fcab16f9700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 0x7fca98032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fca9c0096a0 tx=0x7fca9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:30:58.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.035+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaa8014420 con 0x7fcaac0f51a0 2024-02-13T03:30:58.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.240+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaac0fad30 con 0x7fcaac0f51a0 2024-02-13T03:30:58.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.241+0000 7fcaa27fc700 1 -- 172.21.15.112:0/2949274126 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcaa8010230 con 0x7fcaac0f51a0 2024-02-13T03:30:58.242 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:30:58.242 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:30:58.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 msgr2=0x7fca98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:58.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 0x7fca98032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fca9c0096a0 tx=0x7fca9c006b40).stop 2024-02-13T03:30:58.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 msgr2=0x7fcaac1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:30:58.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fcaa80089f0 tx=0x7fcaa800b040).stop 2024-02-13T03:30:58.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 shutdown_connections 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fca9802ff60 0x7fca98032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 --2- 172.21.15.112:0/2949274126 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcaac0f51a0 0x7fcaac1065d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 >> 172.21.15.112:0/2949274126 conn(0x7fcaac0f09a0 msgr2=0x7fcaac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 shutdown_connections 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:30:58.242+0000 7fcab2efc700 1 -- 172.21.15.112:0/2949274126 wait complete. 2024-02-13T03:30:58.244 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:30:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:56.758921+0000 mgr.smithi080.vyzlfg (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:30:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:30:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:30:58.242249+0000 mon.smithi080 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.112:0/2949274126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:30:59.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:30:59.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:30:58.759232+0000 mgr.smithi080.vyzlfg (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 -- 172.21.15.112:0/2941710204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 msgr2=0x7f16880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 --2- 172.21.15.112:0/2941710204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16880f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1678009fb0 tx=0x7f1678009b30).stop 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 -- 172.21.15.112:0/2941710204 shutdown_connections 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 --2- 172.21.15.112:0/2941710204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16880f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 -- 172.21.15.112:0/2941710204 >> 172.21.15.112:0/2941710204 conn(0x7f16880f0b70 msgr2=0x7f16880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 -- 172.21.15.112:0/2941710204 shutdown_connections 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.713+0000 7f168f5bf700 1 -- 172.21.15.112:0/2941710204 wait complete. 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168f5bf700 1 Processor -- start 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168f5bf700 1 -- start start 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168f5bf700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168f5bf700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1688060280 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168d35b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168d35b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34918/0 (socket says 172.21.15.112:34918) 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168d35b700 1 -- 172.21.15.112:0/3120281603 learned_addr learned my addr 172.21.15.112:0/3120281603 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.714+0000 7f168d35b700 1 -- 172.21.15.112:0/3120281603 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678005040 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f168d35b700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f1678005010 tx=0x7f167800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1678009320 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1688060480 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1678013380 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1678014e50 con 0x7f16880f5410 2024-02-13T03:31:01.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.715+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1688109bd0 con 0x7f16880f5410 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.716+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16780095f0 con 0x7f16880f5410 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.716+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1688045ad0 con 0x7f16880f5410 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.716+0000 7f167dffb700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 0x7f16740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.716+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1678040f40 con 0x7f16880f5410 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.716+0000 7f168cb5a700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 0x7f16740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:01.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.717+0000 7f168cb5a700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 0x7f16740323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f16840096a0 tx=0x7f1684006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.721+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1678014420 con 0x7f16880f5410 2024-02-13T03:31:01.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.926+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16880fafa0 con 0x7f16880f5410 2024-02-13T03:31:01.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.927+0000 7f167dffb700 1 -- 172.21.15.112:0/3120281603 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1678010230 con 0x7f16880f5410 2024-02-13T03:31:01.927 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:01.928 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 msgr2=0x7f16740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 0x7f16740323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f16840096a0 tx=0x7f1684006b40).stop 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 msgr2=0x7f16881094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f1678005010 tx=0x7f167800b040).stop 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 shutdown_connections 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f167402ff10 0x7f16740323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:01.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 --2- 172.21.15.112:0/3120281603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16880f5410 0x7f16881094a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 >> 172.21.15.112:0/3120281603 conn(0x7f16880f0b70 msgr2=0x7f16880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 shutdown_connections 2024-02-13T03:31:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:01.928+0000 7f168f5bf700 1 -- 172.21.15.112:0/3120281603 wait complete. 2024-02-13T03:31:01.930 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:00.759567+0000 mgr.smithi080.vyzlfg (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:01.928001+0000 mon.smithi080 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.112:0/3120281603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:03.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:03.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:02.759836+0000 mgr.smithi080.vyzlfg (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 -- 172.21.15.112:0/3706543810 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 msgr2=0x7f11380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 --2- 172.21.15.112:0/3706543810 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f90f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1128009fb0 tx=0x7f1128009b30).stop 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 -- 172.21.15.112:0/3706543810 shutdown_connections 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 --2- 172.21.15.112:0/3706543810 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f90f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 -- 172.21.15.112:0/3706543810 >> 172.21.15.112:0/3706543810 conn(0x7f11380f0b70 msgr2=0x7f11380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 -- 172.21.15.112:0/3706543810 shutdown_connections 2024-02-13T03:31:05.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.280+0000 7f113fd2f700 1 -- 172.21.15.112:0/3706543810 wait complete. 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.281+0000 7f113fd2f700 1 Processor -- start 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.281+0000 7f113fd2f700 1 -- start start 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.281+0000 7f113fd2f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.281+0000 7f113fd2f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380f4ff0 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113dacb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113dacb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34932/0 (socket says 172.21.15.112:34932) 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113dacb700 1 -- 172.21.15.112:0/1016613346 learned_addr learned my addr 172.21.15.112:0/1016613346 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113dacb700 1 -- 172.21.15.112:0/1016613346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128005040 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113dacb700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1128009830 tx=0x7f112800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1128009320 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11380f5190 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1128013380 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1128014420 con 0x7f1138060280 2024-02-13T03:31:05.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.282+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11380f5620 con 0x7f1138060280 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.283+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f112801e420 con 0x7f1138060280 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.283+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1138059100 con 0x7f1138060280 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.283+0000 7f112e7fc700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 0x7f1124032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.283+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1128040ef0 con 0x7f1138060280 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.284+0000 7f113d2ca700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 0x7f1124032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.284+0000 7f113d2ca700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 0x7f1124032470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f11340096a0 tx=0x7f1134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:05.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.288+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f112801cb10 con 0x7f1138060280 2024-02-13T03:31:05.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.492+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1138045ad0 con 0x7f1138060280 2024-02-13T03:31:05.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.493+0000 7f112e7fc700 1 -- 172.21.15.112:0/1016613346 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1128004490 con 0x7f1138060280 2024-02-13T03:31:05.494 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:05.494 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 msgr2=0x7f1124032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 0x7f1124032470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f11340096a0 tx=0x7f1134006b40).stop 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 msgr2=0x7f11380f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1128009830 tx=0x7f112800b040).stop 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 shutdown_connections 2024-02-13T03:31:05.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f112402ffb0 0x7f1124032470 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:05.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 --2- 172.21.15.112:0/1016613346 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1138060280 0x7f11380f6b70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:05.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 >> 172.21.15.112:0/1016613346 conn(0x7f11380f0b70 msgr2=0x7f11380f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:05.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 shutdown_connections 2024-02-13T03:31:05.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:05.494+0000 7f113fd2f700 1 -- 172.21.15.112:0/1016613346 wait complete. 2024-02-13T03:31:05.496 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:06.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:04.760162+0000 mgr.smithi080.vyzlfg (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:06.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:05.494216+0000 mon.smithi080 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.112:0/1016613346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:07.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:07.134 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:08.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:06.760461+0000 mgr.smithi080.vyzlfg (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:08.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- 172.21.15.112:0/3262128356 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 msgr2=0x7f51f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:08.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/3262128356 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f00f90f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f51e0009fb0 tx=0x7f51e0009b30).stop 2024-02-13T03:31:08.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- 172.21.15.112:0/3262128356 shutdown_connections 2024-02-13T03:31:08.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/3262128356 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f00f90f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- 172.21.15.112:0/3262128356 >> 172.21.15.112:0/3262128356 conn(0x7f51f00f0b70 msgr2=0x7f51f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- 172.21.15.112:0/3262128356 shutdown_connections 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- 172.21.15.112:0/3262128356 wait complete. 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 Processor -- start 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- start start 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.929+0000 7f51f4ef5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f0106c70 con 0x7f51f0060280 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51ee59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51ee59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34958/0 (socket says 172.21.15.112:34958) 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51ee59c700 1 -- 172.21.15.112:0/976058737 learned_addr learned my addr 172.21.15.112:0/976058737 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51ee59c700 1 -- 172.21.15.112:0/976058737 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0005040 con 0x7f51f0060280 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51ee59c700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f51e00089f0 tx=0x7f51e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:08.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e0009320 con 0x7f51f0060280 2024-02-13T03:31:08.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f0106e70 con 0x7f51f0060280 2024-02-13T03:31:08.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51e0013380 con 0x7f51f0060280 2024-02-13T03:31:08.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e0014e20 con 0x7f51f0060280 2024-02-13T03:31:08.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.930+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f0107300 con 0x7f51f0060280 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.931+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f0045ad0 con 0x7f51f0060280 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.931+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51e00095f0 con 0x7f51f0060280 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.931+0000 7f51deffd700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 0x7f51d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.931+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51e0040f00 con 0x7f51f0060280 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.932+0000 7f51edd9b700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 0x7f51d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:08.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.932+0000 7f51edd9b700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 0x7f51d8032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f51e40096a0 tx=0x7f51e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:08.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:08.936+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51e0014420 con 0x7f51f0060280 2024-02-13T03:31:09.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.140+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51f01081b0 con 0x7f51f0060280 2024-02-13T03:31:09.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.141+0000 7f51deffd700 1 -- 172.21.15.112:0/976058737 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51e001d980 con 0x7f51f0060280 2024-02-13T03:31:09.142 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:09.142 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 msgr2=0x7f51d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 0x7f51d8032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f51e40096a0 tx=0x7f51e4006b40).stop 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 msgr2=0x7f51f0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f51e00089f0 tx=0x7f51e000b040).stop 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 shutdown_connections 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f51d802ff60 0x7f51d8032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 --2- 172.21.15.112:0/976058737 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51f0060280 0x7f51f0106780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 >> 172.21.15.112:0/976058737 conn(0x7f51f00f0b70 msgr2=0x7f51f00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 shutdown_connections 2024-02-13T03:31:09.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:09.143+0000 7f51f4ef5700 1 -- 172.21.15.112:0/976058737 wait complete. 2024-02-13T03:31:09.145 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:09.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:09.142608+0000 mon.smithi080 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.112:0/976058737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:10.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:08.760733+0000 mgr.smithi080.vyzlfg (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:10.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:10.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 -- 172.21.15.112:0/2882524819 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 msgr2=0x7f94b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 --2- 172.21.15.112:0/2882524819 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b80f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f94a8009fb0 tx=0x7f94a8009b30).stop 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 -- 172.21.15.112:0/2882524819 shutdown_connections 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 --2- 172.21.15.112:0/2882524819 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b80f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 -- 172.21.15.112:0/2882524819 >> 172.21.15.112:0/2882524819 conn(0x7f94b80f0b70 msgr2=0x7f94b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.593+0000 7f94bf76c700 1 -- 172.21.15.112:0/2882524819 shutdown_connections 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bf76c700 1 -- 172.21.15.112:0/2882524819 wait complete. 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bf76c700 1 Processor -- start 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bf76c700 1 -- start start 2024-02-13T03:31:12.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bf76c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bf76c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b8060280 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bd508700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bd508700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52172/0 (socket says 172.21.15.112:52172) 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.594+0000 7f94bd508700 1 -- 172.21.15.112:0/986928001 learned_addr learned my addr 172.21.15.112:0/986928001 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94bd508700 1 -- 172.21.15.112:0/986928001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8005040 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94bd508700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f94a80089f0 tx=0x7f94a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a8009320 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b8060480 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94a8013380 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a801ed80 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.595+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b8109bd0 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.596+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94a801e420 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.596+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b8045ad0 con 0x7f94b80f5410 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.596+0000 7f94adffb700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 0x7f94a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.596+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94a80410f0 con 0x7f94b80f5410 2024-02-13T03:31:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.596+0000 7f94bcd07700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 0x7f94a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.597+0000 7f94bcd07700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 0x7f94a4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f94b40096a0 tx=0x7f94b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:12.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.601+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94a801cb10 con 0x7f94b80f5410 2024-02-13T03:31:12.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:10.761065+0000 mgr.smithi080.vyzlfg (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:12.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.807+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94b80fafa0 con 0x7f94b80f5410 2024-02-13T03:31:12.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.808+0000 7f94adffb700 1 -- 172.21.15.112:0/986928001 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94a802c2d0 con 0x7f94b80f5410 2024-02-13T03:31:12.810 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:12.810 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 msgr2=0x7f94a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 0x7f94a4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f94b40096a0 tx=0x7f94b4006b40).stop 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 msgr2=0x7f94b81094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f94a80089f0 tx=0x7f94a800b040).stop 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 shutdown_connections 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f94a402ff60 0x7f94a4032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 --2- 172.21.15.112:0/986928001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94b80f5410 0x7f94b81094a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 >> 172.21.15.112:0/986928001 conn(0x7f94b80f0b70 msgr2=0x7f94b80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 shutdown_connections 2024-02-13T03:31:12.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:12.810+0000 7f94bf76c700 1 -- 172.21.15.112:0/986928001 wait complete. 2024-02-13T03:31:12.812 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:13.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:13 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:12.809584+0000 mon.smithi080 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.112:0/986928001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:14.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:14.490 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:14.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:12.761430+0000 mgr.smithi080.vyzlfg (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 -- 172.21.15.112:0/3299515088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 msgr2=0x7f2d1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 --2- 172.21.15.112:0/3299515088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c0f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2d0c009fb0 tx=0x7f2d0c009b30).stop 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 -- 172.21.15.112:0/3299515088 shutdown_connections 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 --2- 172.21.15.112:0/3299515088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c0f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 -- 172.21.15.112:0/3299515088 >> 172.21.15.112:0/3299515088 conn(0x7f2d1c0f1c70 msgr2=0x7f2d1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 -- 172.21.15.112:0/3299515088 shutdown_connections 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.356+0000 7f2d214de700 1 -- 172.21.15.112:0/3299515088 wait complete. 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d214de700 1 Processor -- start 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d214de700 1 -- start start 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d214de700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d214de700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c069e20 con 0x7f2d1c0f6510 2024-02-13T03:31:16.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d1affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d1affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52192/0 (socket says 172.21.15.112:52192) 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.357+0000 7f2d1affd700 1 -- 172.21.15.112:0/1239396363 learned_addr learned my addr 172.21.15.112:0/1239396363 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d1affd700 1 -- 172.21.15.112:0/1239396363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c005040 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d1affd700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d0c005010 tx=0x7f2d0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d0c009320 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c069fc0 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d0c013380 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c06a450 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.358+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d0c014e50 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d0c0095f0 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c059100 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d037fe700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 0x7f2d040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d0c040f00 con 0x7f2d1c0f6510 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d1a7fc700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 0x7f2d040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.359+0000 7f2d1a7fc700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 0x7f2d040323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2d100096a0 tx=0x7f2d10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:16.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.363+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d0c014420 con 0x7f2d1c0f6510 2024-02-13T03:31:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.567+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d1c045ad0 con 0x7f2d1c0f6510 2024-02-13T03:31:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.568+0000 7f2d037fe700 1 -- 172.21.15.112:0/1239396363 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d0c01d9d0 con 0x7f2d1c0f6510 2024-02-13T03:31:16.567 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:16.567 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 msgr2=0x7f2d040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 0x7f2d040323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2d100096a0 tx=0x7f2d10006b40).stop 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 msgr2=0x7f2d1c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d0c005010 tx=0x7f2d0c00b040).stop 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 shutdown_connections 2024-02-13T03:31:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2d0402ff10 0x7f2d040323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:16.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 --2- 172.21.15.112:0/1239396363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2d1c0f6510 0x7f2d1c06b9a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:16.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 >> 172.21.15.112:0/1239396363 conn(0x7f2d1c0f1c70 msgr2=0x7f2d1c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:16.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 shutdown_connections 2024-02-13T03:31:16.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:16.569+0000 7f2d214de700 1 -- 172.21.15.112:0/1239396363 wait complete. 2024-02-13T03:31:16.570 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:16.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:14.761700+0000 mgr.smithi080.vyzlfg (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:17.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:16.567958+0000 mon.smithi080 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.112:0/1239396363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:18.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:18.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:18.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:16.762042+0000 mgr.smithi080.vyzlfg (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 -- 172.21.15.112:0/3357629315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 msgr2=0x7f83c80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 --2- 172.21.15.112:0/3357629315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c80f68a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f83b8009fb0 tx=0x7f83b8009b30).stop 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 -- 172.21.15.112:0/3357629315 shutdown_connections 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 --2- 172.21.15.112:0/3357629315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c80f68a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 -- 172.21.15.112:0/3357629315 >> 172.21.15.112:0/3357629315 conn(0x7f83c80f1c00 msgr2=0x7f83c80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.938+0000 7f83d045d700 1 -- 172.21.15.112:0/3357629315 shutdown_connections 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83d045d700 1 -- 172.21.15.112:0/3357629315 wait complete. 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83d045d700 1 Processor -- start 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83d045d700 1 -- start start 2024-02-13T03:31:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83d045d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83d045d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c8106ce0 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83ce1f9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83ce1f9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52202/0 (socket says 172.21.15.112:52202) 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.939+0000 7f83ce1f9700 1 -- 172.21.15.112:0/2898335043 learned_addr learned my addr 172.21.15.112:0/2898335043 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83ce1f9700 1 -- 172.21.15.112:0/2898335043 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b8005040 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83ce1f9700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f83b8005010 tx=0x7f83b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b8009320 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c8106ee0 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83b8013380 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b8014e50 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.940+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c8107370 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.941+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83b80095f0 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.941+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c8045ad0 con 0x7f83c80f6480 2024-02-13T03:31:19.942 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.941+0000 7f83beffd700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 0x7f83b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.941+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83b8040e70 con 0x7f83c80f6480 2024-02-13T03:31:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.941+0000 7f83cd9f8700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 0x7f83b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.942+0000 7f83cd9f8700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 0x7f83b40323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f83c40096a0 tx=0x7f83c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:19.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:19.946+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83b8014420 con 0x7f83c80f6480 2024-02-13T03:31:20.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.151+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83c80fc090 con 0x7f83c80f6480 2024-02-13T03:31:20.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.152+0000 7f83beffd700 1 -- 172.21.15.112:0/2898335043 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83b8010230 con 0x7f83c80f6480 2024-02-13T03:31:20.152 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:20.152 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 msgr2=0x7f83b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 0x7f83b40323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f83c40096a0 tx=0x7f83c4006b40).stop 2024-02-13T03:31:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 msgr2=0x7f83c81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:20.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f83b8005010 tx=0x7f83b800b040).stop 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 shutdown_connections 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f83b402ff10 0x7f83b40323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 --2- 172.21.15.112:0/2898335043 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83c80f6480 0x7f83c81067f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.153+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 >> 172.21.15.112:0/2898335043 conn(0x7f83c80f1c00 msgr2=0x7f83c80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.154+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 shutdown_connections 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:20.154+0000 7f83d045d700 1 -- 172.21.15.112:0/2898335043 wait complete. 2024-02-13T03:31:20.154 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:20.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:18.762417+0000 mgr.smithi080.vyzlfg (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:20.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:20.152358+0000 mon.smithi080 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.112:0/2898335043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:21.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:21.802 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:22.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:20.762739+0000 mgr.smithi080.vyzlfg (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:23.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/3511894061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 msgr2=0x7fbc140f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/3511894061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc140f9190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fbc10009fb0 tx=0x7fbc10009b30).stop 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/3511894061 shutdown_connections 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/3511894061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc140f9190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/3511894061 >> 172.21.15.112:0/3511894061 conn(0x7fbc140f0b70 msgr2=0x7fbc140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/3511894061 shutdown_connections 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/3511894061 wait complete. 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.614+0000 7fbc1c2d0700 1 Processor -- start 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1c2d0700 1 -- start start 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1c2d0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1c2d0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc14106cc0 con 0x7fbc140601e0 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1a06c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1a06c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:49494/0 (socket says 172.21.15.112:49494) 2024-02-13T03:31:23.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1a06c700 1 -- 172.21.15.112:0/4076469789 learned_addr learned my addr 172.21.15.112:0/4076469789 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1a06c700 1 -- 172.21.15.112:0/4076469789 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10005040 con 0x7fbc140601e0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.615+0000 7fbc1a06c700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbc10007a10 tx=0x7fbc10009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc10018070 con 0x7fbc140601e0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc14106ec0 con 0x7fbc140601e0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc10013410 con 0x7fbc140601e0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc1001c3c0 con 0x7fbc140601e0 2024-02-13T03:31:23.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc14107350 con 0x7fbc140601e0 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc10013ca0 con 0x7fbc140601e0 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.616+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc14045ad0 con 0x7fbc140601e0 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.617+0000 7fbc0affd700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 0x7fbc00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.617+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc10041530 con 0x7fbc140601e0 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.617+0000 7fbc1986b700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 0x7fbc00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:23.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.617+0000 7fbc1986b700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 0x7fbc00032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbc040096a0 tx=0x7fbc04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:23.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.621+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc1001d420 con 0x7fbc140601e0 2024-02-13T03:31:23.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.828+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc14108200 con 0x7fbc140601e0 2024-02-13T03:31:23.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.828+0000 7fbc0affd700 1 -- 172.21.15.112:0/4076469789 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc10025ce0 con 0x7fbc140601e0 2024-02-13T03:31:23.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:23.830 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 msgr2=0x7fbc00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 0x7fbc00032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbc040096a0 tx=0x7fbc04006b40).stop 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 msgr2=0x7fbc141067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbc10007a10 tx=0x7fbc10009b30).stop 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 shutdown_connections 2024-02-13T03:31:23.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbc0002ff60 0x7fbc00032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:23.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 --2- 172.21.15.112:0/4076469789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbc140601e0 0x7fbc141067d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:23.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 >> 172.21.15.112:0/4076469789 conn(0x7fbc140f0b70 msgr2=0x7fbc140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:23.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 shutdown_connections 2024-02-13T03:31:23.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:23.830+0000 7fbc1c2d0700 1 -- 172.21.15.112:0/4076469789 wait complete. 2024-02-13T03:31:23.832 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:24.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:22.763004+0000 mgr.smithi080.vyzlfg (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:24.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:23.829791+0000 mon.smithi080 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.112:0/4076469789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:25.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:25.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:26.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:24.763352+0000 mgr.smithi080.vyzlfg (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/2712046151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 msgr2=0x7f6ae00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/2712046151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae00f55c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f6ad4009fb0 tx=0x7f6ad4009b30).stop 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/2712046151 shutdown_connections 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/2712046151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae00f55c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/2712046151 >> 172.21.15.112:0/2712046151 conn(0x7f6ae00f09a0 msgr2=0x7f6ae00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:27.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/2712046151 shutdown_connections 2024-02-13T03:31:27.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/2712046151 wait complete. 2024-02-13T03:31:27.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.332+0000 7f6ae5bb6700 1 Processor -- start 2024-02-13T03:31:27.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae5bb6700 1 -- start start 2024-02-13T03:31:27.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae5bb6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae5bb6700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01059c0 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae4bb4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae4bb4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:49526/0 (socket says 172.21.15.112:49526) 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.333+0000 7f6ae4bb4700 1 -- 172.21.15.112:0/3006637972 learned_addr learned my addr 172.21.15.112:0/3006637972 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6ae4bb4700 1 -- 172.21.15.112:0/3006637972 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4005040 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6ae4bb4700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6ad4009830 tx=0x7f6ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ad4009320 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae0105bc0 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ad4013380 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ad4014420 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.334+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae0106050 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.335+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ad4014580 con 0x7f6ae00f51a0 2024-02-13T03:31:27.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.335+0000 7f6add7fa700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 0x7f6ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:27.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.335+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ad4040e00 con 0x7f6ae00f51a0 2024-02-13T03:31:27.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.335+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae0045ad0 con 0x7f6ae00f51a0 2024-02-13T03:31:27.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.335+0000 7f6adffff700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 0x7f6ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:27.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.336+0000 7f6adffff700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 0x7f6ad0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6acc0096a0 tx=0x7f6acc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:27.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.340+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ad40273f0 con 0x7f6ae00f51a0 2024-02-13T03:31:27.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.546+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ae0106cd0 con 0x7f6ae00f51a0 2024-02-13T03:31:27.547 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.547+0000 7f6add7fa700 1 -- 172.21.15.112:0/3006637972 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ad401a230 con 0x7f6ae00f51a0 2024-02-13T03:31:27.548 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:27.548 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:27.549 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.548+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 msgr2=0x7f6ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.548+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 0x7f6ad0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f6acc0096a0 tx=0x7f6acc006b40).stop 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.548+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 msgr2=0x7f6ae01054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.548+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6ad4009830 tx=0x7f6ad400b040).stop 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 shutdown_connections 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6ad002ff60 0x7f6ad0032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 --2- 172.21.15.112:0/3006637972 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6ae00f51a0 0x7f6ae01054d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 >> 172.21.15.112:0/3006637972 conn(0x7f6ae00f09a0 msgr2=0x7f6ae00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 shutdown_connections 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:27.549+0000 7f6ae5bb6700 1 -- 172.21.15.112:0/3006637972 wait complete. 2024-02-13T03:31:27.550 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:28.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:26.763707+0000 mgr.smithi080.vyzlfg (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:28.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:27.548333+0000 mon.smithi080 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.112:0/3006637972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:29.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:29.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:30.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:28.764016+0000 mgr.smithi080.vyzlfg (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:30.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/511609095 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 msgr2=0x7f7f040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/511609095 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f040f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f7f00009fb0 tx=0x7f7f00009b30).stop 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/511609095 shutdown_connections 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/511609095 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f040f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/511609095 >> 172.21.15.112:0/511609095 conn(0x7f7f040f0b70 msgr2=0x7f7f040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/511609095 shutdown_connections 2024-02-13T03:31:30.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.973+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/511609095 wait complete. 2024-02-13T03:31:30.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.974+0000 7f7f0c2e5700 1 Processor -- start 2024-02-13T03:31:30.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.974+0000 7f7f0c2e5700 1 -- start start 2024-02-13T03:31:30.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.974+0000 7f7f0c2e5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:30.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.974+0000 7f7f0c2e5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04105bd0 con 0x7f7f040f5410 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0a081700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0a081700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:49542/0 (socket says 172.21.15.112:49542) 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0a081700 1 -- 172.21.15.112:0/3337562648 learned_addr learned my addr 172.21.15.112:0/3337562648 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0a081700 1 -- 172.21.15.112:0/3337562648 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f00005040 con 0x7f7f040f5410 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0a081700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7f00005010 tx=0x7f7f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:30.979 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f00009320 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f04105dd0 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f00013380 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f00014e50 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.975+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f04106260 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.976+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f000095f0 con 0x7f7f040f5410 2024-02-13T03:31:30.980 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.976+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f04045ad0 con 0x7f7f040f5410 2024-02-13T03:31:30.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.976+0000 7f7efaffd700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 0x7f7ef00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:30.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.976+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f00040e90 con 0x7f7f040f5410 2024-02-13T03:31:30.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.977+0000 7f7f09880700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 0x7f7ef00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:30.981 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.977+0000 7f7f09880700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 0x7f7ef00323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7ef40096a0 tx=0x7f7ef4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:30.982 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:30.981+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f00014420 con 0x7f7f040f5410 2024-02-13T03:31:31.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.187+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f04107110 con 0x7f7f040f5410 2024-02-13T03:31:31.188 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.188+0000 7f7efaffd700 1 -- 172.21.15.112:0/3337562648 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f00010230 con 0x7f7f040f5410 2024-02-13T03:31:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:31.189 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 msgr2=0x7f7ef00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 0x7f7ef00323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7ef40096a0 tx=0x7f7ef4006b40).stop 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 msgr2=0x7f7f041056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7f00005010 tx=0x7f7f0000b040).stop 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 shutdown_connections 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7ef002ff10 0x7f7ef00323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 --2- 172.21.15.112:0/3337562648 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7f040f5410 0x7f7f041056e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:31.190 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 >> 172.21.15.112:0/3337562648 conn(0x7f7f040f0b70 msgr2=0x7f7f040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:31.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 shutdown_connections 2024-02-13T03:31:31.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:31.189+0000 7f7f0c2e5700 1 -- 172.21.15.112:0/3337562648 wait complete. 2024-02-13T03:31:31.191 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:31.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:31 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:31.189011+0000 mon.smithi080 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.112:0/3337562648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:32.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:32.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:30.764357+0000 mgr.smithi080.vyzlfg (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 -- 172.21.15.112:0/524494443 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 msgr2=0x7fc7cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/524494443 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc0f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc7c0009fb0 tx=0x7fc7c0009b30).stop 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 -- 172.21.15.112:0/524494443 shutdown_connections 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/524494443 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc0f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 -- 172.21.15.112:0/524494443 >> 172.21.15.112:0/524494443 conn(0x7fc7cc0f1c70 msgr2=0x7fc7cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 -- 172.21.15.112:0/524494443 shutdown_connections 2024-02-13T03:31:34.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.742+0000 7fc7d11f6700 1 -- 172.21.15.112:0/524494443 wait complete. 2024-02-13T03:31:34.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7d11f6700 1 Processor -- start 2024-02-13T03:31:34.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7d11f6700 1 -- start start 2024-02-13T03:31:34.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7d11f6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7d11f6700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc106cd0 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7cad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7cad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53746/0 (socket says 172.21.15.112:53746) 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7cad9d700 1 -- 172.21.15.112:0/2273943719 learned_addr learned my addr 172.21.15.112:0/2273943719 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.743+0000 7fc7cad9d700 1 -- 172.21.15.112:0/2273943719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c0005040 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7cad9d700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc7c0005010 tx=0x7fc7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c0009320 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc106ed0 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7c0013380 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c0014e50 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.744+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc107360 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7c00095f0 con 0x7fc7cc0f6510 2024-02-13T03:31:34.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc045ad0 con 0x7fc7cc0f6510 2024-02-13T03:31:34.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7b37fe700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 0x7fc7b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:34.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7c0040fa0 con 0x7fc7cc0f6510 2024-02-13T03:31:34.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7ca59c700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 0x7fc7b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:34.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.745+0000 7fc7ca59c700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 0x7fc7b40323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc7bc0096a0 tx=0x7fc7bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:34.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.749+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7c0014420 con 0x7fc7cc0f6510 2024-02-13T03:31:34.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:32.764715+0000 mgr.smithi080.vyzlfg (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.954+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7cc108210 con 0x7fc7cc0f6510 2024-02-13T03:31:34.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.955+0000 7fc7b37fe700 1 -- 172.21.15.112:0/2273943719 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7c0010230 con 0x7fc7cc0f6510 2024-02-13T03:31:34.956 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:34.956 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:34.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 msgr2=0x7fc7b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 0x7fc7b40323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc7bc0096a0 tx=0x7fc7bc006b40).stop 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 msgr2=0x7fc7cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc7c0005010 tx=0x7fc7c000b040).stop 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 shutdown_connections 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc7b402ff10 0x7fc7b40323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 --2- 172.21.15.112:0/2273943719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc7cc0f6510 0x7fc7cc1067e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 >> 172.21.15.112:0/2273943719 conn(0x7fc7cc0f1c70 msgr2=0x7fc7cc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 shutdown_connections 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:34.957+0000 7fc7d11f6700 1 -- 172.21.15.112:0/2273943719 wait complete. 2024-02-13T03:31:34.958 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:35.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:35 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:34.956249+0000 mon.smithi080 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.112:0/2273943719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:36.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:36.650 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:34.765100+0000 mgr.smithi080.vyzlfg (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 -- 172.21.15.112:0/4185457171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 msgr2=0x7f792c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 --2- 172.21.15.112:0/4185457171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c0f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7914009fb0 tx=0x7f7914009b30).stop 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 -- 172.21.15.112:0/4185457171 shutdown_connections 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 --2- 172.21.15.112:0/4185457171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c0f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 -- 172.21.15.112:0/4185457171 >> 172.21.15.112:0/4185457171 conn(0x7f792c0f1c70 msgr2=0x7f792c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 -- 172.21.15.112:0/4185457171 shutdown_connections 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.526+0000 7f793090e700 1 -- 172.21.15.112:0/4185457171 wait complete. 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f793090e700 1 Processor -- start 2024-02-13T03:31:38.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f793090e700 1 -- start start 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f793090e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f793090e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c069e20 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f792a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f792a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:53760/0 (socket says 172.21.15.112:53760) 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.527+0000 7f792a59c700 1 -- 172.21.15.112:0/4064944845 learned_addr learned my addr 172.21.15.112:0/4064944845 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f792a59c700 1 -- 172.21.15.112:0/4064944845 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7914005040 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f792a59c700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f79140089f0 tx=0x7f791400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7914009320 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c069fc0 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7914013380 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791401ed80 con 0x7f792c0f6510 2024-02-13T03:31:38.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.528+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c06a450 con 0x7f792c0f6510 2024-02-13T03:31:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f791401e420 con 0x7f792c0f6510 2024-02-13T03:31:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c059100 con 0x7f792c0f6510 2024-02-13T03:31:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f7922ffd700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 0x7f7918032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:38.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79140410e0 con 0x7f792c0f6510 2024-02-13T03:31:38.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f7929d9b700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 0x7f7918032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:38.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.529+0000 7f7929d9b700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 0x7f7918032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f791c0096a0 tx=0x7f791c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:38.534 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.533+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f791401cb10 con 0x7f792c0f6510 2024-02-13T03:31:38.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.738+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f792c045ad0 con 0x7f792c0f6510 2024-02-13T03:31:38.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.738+0000 7f7922ffd700 1 -- 172.21.15.112:0/4064944845 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f791402c2d0 con 0x7f792c0f6510 2024-02-13T03:31:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:38.739 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 msgr2=0x7f7918032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 0x7f7918032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f791c0096a0 tx=0x7f791c006b40).stop 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 msgr2=0x7f792c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f79140089f0 tx=0x7f791400b040).stop 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 shutdown_connections 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f791802ff60 0x7f7918032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 --2- 172.21.15.112:0/4064944845 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f792c0f6510 0x7f792c06b9a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 >> 172.21.15.112:0/4064944845 conn(0x7f792c0f1c70 msgr2=0x7f792c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 shutdown_connections 2024-02-13T03:31:38.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:38.740+0000 7f793090e700 1 -- 172.21.15.112:0/4064944845 wait complete. 2024-02-13T03:31:38.742 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:36.765343+0000 mgr.smithi080.vyzlfg (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:39.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:38.739709+0000 mon.smithi080 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.112:0/4064944845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:40.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:40.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:38.765603+0000 mgr.smithi080.vyzlfg (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- 172.21.15.112:0/2267848604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 msgr2=0x7f4bd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 --2- 172.21.15.112:0/2267848604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd00f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f4bc0009fb0 tx=0x7f4bc0009b30).stop 2024-02-13T03:31:42.206 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- 172.21.15.112:0/2267848604 shutdown_connections 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 --2- 172.21.15.112:0/2267848604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd00f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- 172.21.15.112:0/2267848604 >> 172.21.15.112:0/2267848604 conn(0x7f4bd00f0b70 msgr2=0x7f4bd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- 172.21.15.112:0/2267848604 shutdown_connections 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- 172.21.15.112:0/2267848604 wait complete. 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 Processor -- start 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- start start 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.204+0000 7f4bd562b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd0105c10 con 0x7f4bd00f5410 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bceffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bceffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:41778/0 (socket says 172.21.15.112:41778) 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bceffd700 1 -- 172.21.15.112:0/3968894151 learned_addr learned my addr 172.21.15.112:0/3968894151 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bceffd700 1 -- 172.21.15.112:0/3968894151 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0005040 con 0x7f4bd00f5410 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bceffd700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4bc0005010 tx=0x7f4bc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:42.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc0009320 con 0x7f4bd00f5410 2024-02-13T03:31:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd0105e10 con 0x7f4bd00f5410 2024-02-13T03:31:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bc0013380 con 0x7f4bd00f5410 2024-02-13T03:31:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc001ce70 con 0x7f4bd00f5410 2024-02-13T03:31:42.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.205+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd01062a0 con 0x7f4bd00f5410 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.206+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bc001c440 con 0x7f4bd00f5410 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.206+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd0045ad0 con 0x7f4bd00f5410 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.206+0000 7f4bb77fe700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 0x7f4bb8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.206+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bc0042f20 con 0x7f4bd00f5410 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.207+0000 7f4bce7fc700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 0x7f4bb8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:42.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.207+0000 7f4bce7fc700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 0x7f4bb8032370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4bc40096a0 tx=0x7f4bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.211+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bc00095f0 con 0x7f4bd00f5410 2024-02-13T03:31:42.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.415+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bd01070f0 con 0x7f4bd00f5410 2024-02-13T03:31:42.416 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.416+0000 7f4bb77fe700 1 -- 172.21.15.112:0/3968894151 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bc0012330 con 0x7f4bd00f5410 2024-02-13T03:31:42.417 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:42.417 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 msgr2=0x7f4bb8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 0x7f4bb8032370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4bc40096a0 tx=0x7f4bc4006b40).stop 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 msgr2=0x7f4bd0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4bc0005010 tx=0x7f4bc000b040).stop 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 shutdown_connections 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4bb802feb0 0x7f4bb8032370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 --2- 172.21.15.112:0/3968894151 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd00f5410 0x7f4bd0105720 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 >> 172.21.15.112:0/3968894151 conn(0x7f4bd00f0b70 msgr2=0x7f4bd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 shutdown_connections 2024-02-13T03:31:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:42.417+0000 7f4bd562b700 1 -- 172.21.15.112:0/3968894151 wait complete. 2024-02-13T03:31:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:40.765995+0000 mgr.smithi080.vyzlfg (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:42.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:42.417058+0000 mon.smithi080 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.112:0/3968894151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:44.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:44.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:42.766361+0000 mgr.smithi080.vyzlfg (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 -- 172.21.15.112:0/3314542865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 msgr2=0x7fd5cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/3314542865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc0f55c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fd5bc009fb0 tx=0x7fd5bc009b30).stop 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 -- 172.21.15.112:0/3314542865 shutdown_connections 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/3314542865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc0f55c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 -- 172.21.15.112:0/3314542865 >> 172.21.15.112:0/3314542865 conn(0x7fd5cc0f09a0 msgr2=0x7fd5cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 -- 172.21.15.112:0/3314542865 shutdown_connections 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.926+0000 7fd5d0d45700 1 -- 172.21.15.112:0/3314542865 wait complete. 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5d0d45700 1 Processor -- start 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5d0d45700 1 -- start start 2024-02-13T03:31:45.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5d0d45700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5d0d45700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc108cc0 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:41796/0 (socket says 172.21.15.112:41796) 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.927+0000 7fd5cb7fe700 1 -- 172.21.15.112:0/2417884008 learned_addr learned my addr 172.21.15.112:0/2417884008 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5cb7fe700 1 -- 172.21.15.112:0/2417884008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc005040 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5cb7fe700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd5bc0089f0 tx=0x7fd5bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5bc009320 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc108ec0 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5bc013380 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5bc014e20 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.928+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc109350 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.929+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5bc0263a0 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.929+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc045ad0 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.929+0000 7fd5b3fff700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:45.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.929+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5bc040ee0 con 0x7fd5cc0f51a0 2024-02-13T03:31:45.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.929+0000 7fd5caffd700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:45.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.930+0000 7fd5caffd700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 0x7fd5b40323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd5c00096a0 tx=0x7fd5c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:45.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:45.934+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5bc0126f0 con 0x7fd5cc0f51a0 2024-02-13T03:31:46.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.138+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5cc10a200 con 0x7fd5cc0f51a0 2024-02-13T03:31:46.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.139+0000 7fd5b3fff700 1 -- 172.21.15.112:0/2417884008 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5bc0126f0 con 0x7fd5cc0f51a0 2024-02-13T03:31:46.139 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:46.140 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 msgr2=0x7fd5b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 0x7fd5b40323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd5c00096a0 tx=0x7fd5c0006b40).stop 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 msgr2=0x7fd5cc1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd5bc0089f0 tx=0x7fd5bc00b040).stop 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 shutdown_connections 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 --2- 172.21.15.112:0/2417884008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5cc0f51a0 0x7fd5cc1087d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.140+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 >> 172.21.15.112:0/2417884008 conn(0x7fd5cc0f09a0 msgr2=0x7fd5cc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.141+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 shutdown_connections 2024-02-13T03:31:46.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:46.141+0000 7fd5d0d45700 1 -- 172.21.15.112:0/2417884008 wait complete. 2024-02-13T03:31:46.143 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:44.766655+0000 mgr.smithi080.vyzlfg (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:46.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:46.139839+0000 mon.smithi080 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.112:0/2417884008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:47.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:47.722 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:46.766911+0000 mgr.smithi080.vyzlfg (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- 172.21.15.112:0/4293011949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 msgr2=0x7f0f2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/4293011949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f0f1c009fb0 tx=0x7f0f1c009b30).stop 2024-02-13T03:31:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- 172.21.15.112:0/4293011949 shutdown_connections 2024-02-13T03:31:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/4293011949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- 172.21.15.112:0/4293011949 >> 172.21.15.112:0/4293011949 conn(0x7f0f2c0f0b70 msgr2=0x7f0f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- 172.21.15.112:0/4293011949 shutdown_connections 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- 172.21.15.112:0/4293011949 wait complete. 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 Processor -- start 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- start start 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.577+0000 7f0f30ad3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0fb960 con 0x7f0f2c0f5410 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f2a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f2a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:41812/0 (socket says 172.21.15.112:41812) 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f2a59c700 1 -- 172.21.15.112:0/1239511936 learned_addr learned my addr 172.21.15.112:0/1239511936 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f2a59c700 1 -- 172.21.15.112:0/1239511936 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c005040 con 0x7f0f2c0f5410 2024-02-13T03:31:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f2a59c700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0f1c005010 tx=0x7f0f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c009320 con 0x7f0f2c0f5410 2024-02-13T03:31:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c0fbb00 con 0x7f0f2c0f5410 2024-02-13T03:31:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f1c013380 con 0x7f0f2c0f5410 2024-02-13T03:31:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1c014e50 con 0x7f0f2c0f5410 2024-02-13T03:31:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.578+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c0fbf90 con 0x7f0f2c0f5410 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.579+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f1c014420 con 0x7f0f2c0f5410 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.579+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c059100 con 0x7f0f2c0f5410 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.579+0000 7f0f22ffd700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.579+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f1c040f70 con 0x7f0f2c0f5410 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.579+0000 7f0f29d9b700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.580+0000 7f0f29d9b700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:49.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.584+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f1c0095f0 con 0x7f0f2c0f5410 2024-02-13T03:31:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.786+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f2c045ad0 con 0x7f0f2c0f5410 2024-02-13T03:31:49.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.787+0000 7f0f22ffd700 1 -- 172.21.15.112:0/1239511936 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f1c01db70 con 0x7f0f2c0f5410 2024-02-13T03:31:49.788 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:49.788 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 msgr2=0x7f0f180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).stop 2024-02-13T03:31:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 msgr2=0x7f0f2c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0f1c005010 tx=0x7f0f1c00b040).stop 2024-02-13T03:31:49.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 shutdown_connections 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 --2- 172.21.15.112:0/1239511936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f2c0f5410 0x7f0f2c0fd4e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.788+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 >> 172.21.15.112:0/1239511936 conn(0x7f0f2c0f0b70 msgr2=0x7f0f2c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.789+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 shutdown_connections 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:49.789+0000 7f0f30ad3700 1 -- 172.21.15.112:0/1239511936 wait complete. 2024-02-13T03:31:49.790 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:48.767260+0000 mgr.smithi080.vyzlfg (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:50.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:49.788216+0000 mon.smithi080 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.112:0/1239511936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:51.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:51.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:50.767638+0000 mgr.smithi080.vyzlfg (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:53.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 -- 172.21.15.112:0/3771859491 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 msgr2=0x7fb34c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:53.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 --2- 172.21.15.112:0/3771859491 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c0f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb334009fb0 tx=0x7fb334009b30).stop 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 -- 172.21.15.112:0/3771859491 shutdown_connections 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 --2- 172.21.15.112:0/3771859491 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c0f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 -- 172.21.15.112:0/3771859491 >> 172.21.15.112:0/3771859491 conn(0x7fb34c0f0b70 msgr2=0x7fb34c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 -- 172.21.15.112:0/3771859491 shutdown_connections 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.358+0000 7fb350bf4700 1 -- 172.21.15.112:0/3771859491 wait complete. 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb350bf4700 1 Processor -- start 2024-02-13T03:31:53.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb350bf4700 1 -- start start 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb350bf4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb350bf4700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c060240 con 0x7fb34c0f5410 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb34a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb34a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37578/0 (socket says 172.21.15.112:37578) 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.359+0000 7fb34a59c700 1 -- 172.21.15.112:0/705746977 learned_addr learned my addr 172.21.15.112:0/705746977 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb34a59c700 1 -- 172.21.15.112:0/705746977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334005040 con 0x7fb34c0f5410 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb34a59c700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb334005010 tx=0x7fb33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb334009320 con 0x7fb34c0f5410 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb334013380 con 0x7fb34c0f5410 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c060440 con 0x7fb34c0f5410 2024-02-13T03:31:53.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb334014e50 con 0x7fb34c0f5410 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.360+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c109bc0 con 0x7fb34c0f5410 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb334014420 con 0x7fb34c0f5410 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c045ad0 con 0x7fb34c0f5410 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb342ffd700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb334040f40 con 0x7fb34c0f5410 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb349d9b700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.361+0000 7fb349d9b700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 0x7fb3380323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.366+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3340095f0 con 0x7fb34c0f5410 2024-02-13T03:31:53.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.570+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb34c10a3e0 con 0x7fb34c0f5410 2024-02-13T03:31:53.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.571+0000 7fb342ffd700 1 -- 172.21.15.112:0/705746977 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb33401db70 con 0x7fb34c0f5410 2024-02-13T03:31:53.571 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:53.571 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:53.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 msgr2=0x7fb3380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:53.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 0x7fb3380323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb33c0096a0 tx=0x7fb33c006b40).stop 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 msgr2=0x7fb34c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb334005010 tx=0x7fb33400b040).stop 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 shutdown_connections 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 --2- 172.21.15.112:0/705746977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb34c0f5410 0x7fb34c109510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 >> 172.21.15.112:0/705746977 conn(0x7fb34c0f0b70 msgr2=0x7fb34c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 shutdown_connections 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:53.572+0000 7fb350bf4700 1 -- 172.21.15.112:0/705746977 wait complete. 2024-02-13T03:31:53.573 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:52.768022+0000 mgr.smithi080.vyzlfg (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:53.570904+0000 mon.smithi080 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.112:0/705746977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:55.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:55.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:54.768391+0000 mgr.smithi080.vyzlfg (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:31:57.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 -- 172.21.15.112:0/2601488863 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 msgr2=0x7f549c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:57.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 --2- 172.21.15.112:0/2601488863 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f548c009fb0 tx=0x7f548c009b30).stop 2024-02-13T03:31:57.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 -- 172.21.15.112:0/2601488863 shutdown_connections 2024-02-13T03:31:57.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 --2- 172.21.15.112:0/2601488863 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 -- 172.21.15.112:0/2601488863 >> 172.21.15.112:0/2601488863 conn(0x7f549c0f0b70 msgr2=0x7f549c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 -- 172.21.15.112:0/2601488863 shutdown_connections 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.136+0000 7f54a39e9700 1 -- 172.21.15.112:0/2601488863 wait complete. 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a39e9700 1 Processor -- start 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a39e9700 1 -- start start 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a39e9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a39e9700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c060240 con 0x7f549c0f5410 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a1785700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a1785700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37604/0 (socket says 172.21.15.112:37604) 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a1785700 1 -- 172.21.15.112:0/1007305298 learned_addr learned my addr 172.21.15.112:0/1007305298 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.137+0000 7f54a1785700 1 -- 172.21.15.112:0/1007305298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c005040 con 0x7f549c0f5410 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f54a1785700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f548c005010 tx=0x7f548c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548c009320 con 0x7f549c0f5410 2024-02-13T03:31:57.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c060440 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f548c013380 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548c014e50 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.138+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c109b90 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.139+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f548c0095f0 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.139+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c045ad0 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.139+0000 7f5491ffb700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 0x7f54880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.139+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f548c040f80 con 0x7f549c0f5410 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.139+0000 7f54a0f84700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 0x7f54880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.140+0000 7f54a0f84700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 0x7f54880323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f54980096a0 tx=0x7f5498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:31:57.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.143+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f548c014420 con 0x7f549c0f5410 2024-02-13T03:31:57.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.349+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f549c10a3b0 con 0x7f549c0f5410 2024-02-13T03:31:57.349 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.349+0000 7f5491ffb700 1 -- 172.21.15.112:0/1007305298 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f548c010230 con 0x7f549c0f5410 2024-02-13T03:31:57.349 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:31:57.349 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 msgr2=0x7f54880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 0x7f54880323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f54980096a0 tx=0x7f5498006b40).stop 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 msgr2=0x7f549c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f548c005010 tx=0x7f548c00b040).stop 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 shutdown_connections 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f548802ff10 0x7f54880323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 --2- 172.21.15.112:0/1007305298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c0f5410 0x7f549c1094e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 >> 172.21.15.112:0/1007305298 conn(0x7f549c0f0b70 msgr2=0x7f549c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 shutdown_connections 2024-02-13T03:31:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:31:57.351+0000 7f54a39e9700 1 -- 172.21.15.112:0/1007305298 wait complete. 2024-02-13T03:31:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:31:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:56.858020+0000 mon.smithi080 (mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:31:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:56.865812+0000 mon.smithi080 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:31:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:31:57.349752+0000 mon.smithi080 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.112:0/1007305298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:31:58.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:31:58.937 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:31:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:31:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:56.768732+0000 mgr.smithi080.vyzlfg (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:00.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 -- 172.21.15.112:0/3044382537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 msgr2=0x7f25980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:00.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 --2- 172.21.15.112:0/3044382537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25980f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f258c009fb0 tx=0x7f258c009b30).stop 2024-02-13T03:32:00.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 -- 172.21.15.112:0/3044382537 shutdown_connections 2024-02-13T03:32:00.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 --2- 172.21.15.112:0/3044382537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25980f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 -- 172.21.15.112:0/3044382537 >> 172.21.15.112:0/3044382537 conn(0x7f25980f0b70 msgr2=0x7f25980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 -- 172.21.15.112:0/3044382537 shutdown_connections 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.759+0000 7f259c963700 1 -- 172.21.15.112:0/3044382537 wait complete. 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259c963700 1 Processor -- start 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259c963700 1 -- start start 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259c963700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259c963700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2598060240 con 0x7f25980f5410 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37626/0 (socket says 172.21.15.112:37626) 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259659c700 1 -- 172.21.15.112:0/951676968 learned_addr learned my addr 172.21.15.112:0/951676968 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259659c700 1 -- 172.21.15.112:0/951676968 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c005040 con 0x7f25980f5410 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.760+0000 7f259659c700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f258c005010 tx=0x7f258c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.761+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258c009320 con 0x7f25980f5410 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.761+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2598060440 con 0x7f25980f5410 2024-02-13T03:32:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.761+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f258c013380 con 0x7f25980f5410 2024-02-13T03:32:00.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.761+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f258c014e50 con 0x7f25980f5410 2024-02-13T03:32:00.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.761+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2598109b90 con 0x7f25980f5410 2024-02-13T03:32:00.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f258c0095f0 con 0x7f25980f5410 2024-02-13T03:32:00.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2598045ad0 con 0x7f25980f5410 2024-02-13T03:32:00.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f2586ffd700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 0x7f25800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:00.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f258c040e90 con 0x7f25980f5410 2024-02-13T03:32:00.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f2595d9b700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 0x7f25800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:00.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.762+0000 7f2595d9b700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 0x7f25800323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f25880096a0 tx=0x7f2588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:00.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.767+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f258c014420 con 0x7f25980f5410 2024-02-13T03:32:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:31:58.769064+0000 mgr.smithi080.vyzlfg (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:00.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.973+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f259810a3b0 con 0x7f25980f5410 2024-02-13T03:32:00.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.973+0000 7f2586ffd700 1 -- 172.21.15.112:0/951676968 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f258c010230 con 0x7f25980f5410 2024-02-13T03:32:00.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:00.973 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 msgr2=0x7f25800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 0x7f25800323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f25880096a0 tx=0x7f2588006b40).stop 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 msgr2=0x7f25981094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f258c005010 tx=0x7f258c00b040).stop 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 shutdown_connections 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f258002ff10 0x7f25800323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 --2- 172.21.15.112:0/951676968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f25980f5410 0x7f25981094e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 >> 172.21.15.112:0/951676968 conn(0x7f25980f0b70 msgr2=0x7f25980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 shutdown_connections 2024-02-13T03:32:00.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:00.975+0000 7f259c963700 1 -- 172.21.15.112:0/951676968 wait complete. 2024-02-13T03:32:00.976 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:01.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:01 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:00.973807+0000 mon.smithi080 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.112:0/951676968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:02.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:02.630 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:00.769399+0000 mgr.smithi080.vyzlfg (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 -- 172.21.15.112:0/2769901743 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 msgr2=0x7f6a200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 --2- 172.21.15.112:0/2769901743 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a200f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f6a14009fb0 tx=0x7f6a14009b30).stop 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 -- 172.21.15.112:0/2769901743 shutdown_connections 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 --2- 172.21.15.112:0/2769901743 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a200f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 -- 172.21.15.112:0/2769901743 >> 172.21.15.112:0/2769901743 conn(0x7f6a200f0b70 msgr2=0x7f6a200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 -- 172.21.15.112:0/2769901743 shutdown_connections 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.535+0000 7f6a259e7700 1 -- 172.21.15.112:0/2769901743 wait complete. 2024-02-13T03:32:04.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a259e7700 1 Processor -- start 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a259e7700 1 -- start start 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a259e7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a259e7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20105c10 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a1effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a1effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:35686/0 (socket says 172.21.15.112:35686) 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a1effd700 1 -- 172.21.15.112:0/836712494 learned_addr learned my addr 172.21.15.112:0/836712494 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.536+0000 7f6a1effd700 1 -- 172.21.15.112:0/836712494 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14005040 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a1effd700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6a14005010 tx=0x7f6a1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a14009320 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a20105e10 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a14013380 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a14014e50 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.537+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a201062a0 con 0x7f6a200f5410 2024-02-13T03:32:04.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a14014420 con 0x7f6a200f5410 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a077fe700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 0x7f6a080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20045ad0 con 0x7f6a200f5410 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a14040e30 con 0x7f6a200f5410 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a1e7fc700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 0x7f6a080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.538+0000 7f6a1e7fc700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 0x7f6a080323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6a100096a0 tx=0x7f6a10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:04.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.543+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a140095f0 con 0x7f6a200f5410 2024-02-13T03:32:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.746+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a20107150 con 0x7f6a200f5410 2024-02-13T03:32:04.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.746+0000 7f6a077fe700 1 -- 172.21.15.112:0/836712494 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a1402c090 con 0x7f6a200f5410 2024-02-13T03:32:04.747 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:04.747 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 msgr2=0x7f6a080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 0x7f6a080323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6a100096a0 tx=0x7f6a10006b40).stop 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 msgr2=0x7f6a20105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f6a14005010 tx=0x7f6a1400b040).stop 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 shutdown_connections 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6a0802ff10 0x7f6a080323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 --2- 172.21.15.112:0/836712494 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a200f5410 0x7f6a20105720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 >> 172.21.15.112:0/836712494 conn(0x7f6a200f0b70 msgr2=0x7f6a200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 shutdown_connections 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:04.748+0000 7f6a259e7700 1 -- 172.21.15.112:0/836712494 wait complete. 2024-02-13T03:32:04.749 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:04.906 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:02.769678+0000 mgr.smithi080.vyzlfg (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:05.662 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:04.747756+0000 mon.smithi080 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.112:0/836712494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:05.662 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:04.908234+0000 mon.smithi080 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi080", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-13T03:32:05.662 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:04.911733+0000 mon.smithi080 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi080", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-13T03:32:05.662 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:04.912504+0000 mon.smithi080 (mon.0) 201 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T03:32:06.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:06.346 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:04.770029+0000 mgr.smithi080.vyzlfg (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:32:04.906969+0000 mgr.smithi080.vyzlfg (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi080 ... 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:27:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:28:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:28:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.692 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:30:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:30:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:31:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-13T03:32:06.693 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('uid/gid not found') 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: uid/gid not found 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: yield (conn, connr) 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: code, '\n'.join(err))) 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi080 ... 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-13T03:32:06.694 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:27:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:28:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:28:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:30:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:30:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:31:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:32:06.695 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('uid/gid not found') 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: uid/gid not found 2024-02-13T03:32:06.696 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:32:04.913199+0000 mgr.smithi080.vyzlfg (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi080 on smithi080 2024-02-13T03:32:08.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.238+0000 7f35b7fff700 1 -- 172.21.15.112:0/1556282531 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 msgr2=0x7f35b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:08.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.238+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1556282531 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b80f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f35a8009fb0 tx=0x7f35a8009b30).stop 2024-02-13T03:32:08.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 -- 172.21.15.112:0/1556282531 shutdown_connections 2024-02-13T03:32:08.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1556282531 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b80f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:08.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 -- 172.21.15.112:0/1556282531 >> 172.21.15.112:0/1556282531 conn(0x7f35b80f09a0 msgr2=0x7f35b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 -- 172.21.15.112:0/1556282531 shutdown_connections 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 -- 172.21.15.112:0/1556282531 wait complete. 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 Processor -- start 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.239+0000 7f35b7fff700 1 -- start start 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b7fff700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81059c0 con 0x7f35b80f51a0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b6ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b6ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:35706/0 (socket says 172.21.15.112:35706) 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b6ffd700 1 -- 172.21.15.112:0/1517634358 learned_addr learned my addr 172.21.15.112:0/1517634358 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b6ffd700 1 -- 172.21.15.112:0/1517634358 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8005040 con 0x7f35b80f51a0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.240+0000 7f35b6ffd700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f35a80089f0 tx=0x7f35a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35a8009320 con 0x7f35b80f51a0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35a8013380 con 0x7f35b80f51a0 2024-02-13T03:32:08.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b8105bc0 con 0x7f35b80f51a0 2024-02-13T03:32:08.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35a8014e20 con 0x7f35b80f51a0 2024-02-13T03:32:08.243 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b8106050 con 0x7f35b80f51a0 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.241+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35a8014420 con 0x7f35b80f51a0 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.242+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8045ad0 con 0x7f35b80f51a0 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.242+0000 7f359f7fe700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 0x7f35a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.242+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35a8040ed0 con 0x7f35b80f51a0 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.242+0000 7f35b67fc700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 0x7f35a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:08.244 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.242+0000 7f35b67fc700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 0x7f35a00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f35ac0096a0 tx=0x7f35ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:08.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.246+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35a8014790 con 0x7f35b80f51a0 2024-02-13T03:32:08.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.450+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35b8106ea0 con 0x7f35b80f51a0 2024-02-13T03:32:08.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.450+0000 7f359f7fe700 1 -- 172.21.15.112:0/1517634358 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35a801d7f0 con 0x7f35b80f51a0 2024-02-13T03:32:08.451 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:08.451 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:08.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 msgr2=0x7f35a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 0x7f35a00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f35ac0096a0 tx=0x7f35ac006b40).stop 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 msgr2=0x7f35b81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f35a80089f0 tx=0x7f35a800b040).stop 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 shutdown_connections 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f35a002ff10 0x7f35a00323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 --2- 172.21.15.112:0/1517634358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f35b80f51a0 0x7f35b81054d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 >> 172.21.15.112:0/1517634358 conn(0x7f35b80f09a0 msgr2=0x7f35b80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 shutdown_connections 2024-02-13T03:32:08.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:08.452+0000 7f35b7fff700 1 -- 172.21.15.112:0/1517634358 wait complete. 2024-02-13T03:32:08.454 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:08.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:06.770329+0000 mgr.smithi080.vyzlfg (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.250881+0000 mon.smithi080 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.418600+0000 mon.smithi080 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.420791+0000 mon.smithi080 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.421220+0000 mon.smithi080 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.421632+0000 mgr.smithi080.vyzlfg (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:07.423881+0000 mon.smithi080 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:32:08.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:32:07.426634+0000 mgr.smithi080.vyzlfg (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi080 on smithi080 2024-02-13T03:32:09.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:08.451855+0000 mon.smithi080 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.112:0/1517634358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:10.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:10.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:10.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:08.770678+0000 mgr.smithi080.vyzlfg (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 -- 172.21.15.112:0/3025524503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 msgr2=0x7f6da00f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/3025524503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da00f6e30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6d94009fb0 tx=0x7f6d94009b30).stop 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 -- 172.21.15.112:0/3025524503 shutdown_connections 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/3025524503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da00f6e30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 -- 172.21.15.112:0/3025524503 >> 172.21.15.112:0/3025524503 conn(0x7f6da00f02e0 msgr2=0x7f6da00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 -- 172.21.15.112:0/3025524503 shutdown_connections 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.967+0000 7f6da5e1d700 1 -- 172.21.15.112:0/3025524503 wait complete. 2024-02-13T03:32:11.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.968+0000 7f6da5e1d700 1 Processor -- start 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.968+0000 7f6da5e1d700 1 -- start start 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.968+0000 7f6da5e1d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.968+0000 7f6da5e1d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0106c90 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.968+0000 7f6d9f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d9f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46866/0 (socket says 172.21.15.112:46866) 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d9f7fe700 1 -- 172.21.15.112:0/1307915306 learned_addr learned my addr 172.21.15.112:0/1307915306 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d9f7fe700 1 -- 172.21.15.112:0/1307915306 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94005040 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d9f7fe700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6d94005010 tx=0x7f6d9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d94009320 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da0106e90 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d94013380 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d94014e50 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.969+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da0107320 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.970+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d940263a0 con 0x7f6da00f6a10 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.970+0000 7f6d87fff700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:11.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.970+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d94040eb0 con 0x7f6da00f6a10 2024-02-13T03:32:11.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.970+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da0045ad0 con 0x7f6da00f6a10 2024-02-13T03:32:11.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.970+0000 7f6d9effd700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:11.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.971+0000 7f6d9effd700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6d900096a0 tx=0x7f6d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:11.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:11.975+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d94012920 con 0x7f6da00f6a10 2024-02-13T03:32:12.183 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.182+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6da0108060 con 0x7f6da00f6a10 2024-02-13T03:32:12.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.183+0000 7f6d87fff700 1 -- 172.21.15.112:0/1307915306 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d9402b020 con 0x7f6da00f6a10 2024-02-13T03:32:12.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:12.184 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:12.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 msgr2=0x7f6d880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6d900096a0 tx=0x7f6d90006b40).stop 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 msgr2=0x7f6da01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6d94005010 tx=0x7f6d9400b040).stop 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 shutdown_connections 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f6d8802ff10 0x7f6d880323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 --2- 172.21.15.112:0/1307915306 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6da00f6a10 0x7f6da01067a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 >> 172.21.15.112:0/1307915306 conn(0x7f6da00f02e0 msgr2=0x7f6da00f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 shutdown_connections 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:12.185+0000 7f6da5e1d700 1 -- 172.21.15.112:0/1307915306 wait complete. 2024-02-13T03:32:12.186 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:12.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:10.770979+0000 mgr.smithi080.vyzlfg (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:12.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:12.184439+0000 mon.smithi080 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.112:0/1307915306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:13.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:13.759 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:14.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:12.771312+0000 mgr.smithi080.vyzlfg (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:15.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 -- 172.21.15.112:0/1172878311 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 msgr2=0x7f48cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:15.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 --2- 172.21.15.112:0/1172878311 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc0f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f48bc009fb0 tx=0x7f48bc009b30).stop 2024-02-13T03:32:15.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 -- 172.21.15.112:0/1172878311 shutdown_connections 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 --2- 172.21.15.112:0/1172878311 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc0f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 -- 172.21.15.112:0/1172878311 >> 172.21.15.112:0/1172878311 conn(0x7f48cc0f0b70 msgr2=0x7f48cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 -- 172.21.15.112:0/1172878311 shutdown_connections 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 -- 172.21.15.112:0/1172878311 wait complete. 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.576+0000 7f48d3785700 1 Processor -- start 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d3785700 1 -- start start 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d3785700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d3785700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc105bd0 con 0x7f48cc0f5410 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d1521700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d1521700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46886/0 (socket says 172.21.15.112:46886) 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d1521700 1 -- 172.21.15.112:0/3891037265 learned_addr learned my addr 172.21.15.112:0/3891037265 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d1521700 1 -- 172.21.15.112:0/3891037265 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc005040 con 0x7f48cc0f5410 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d1521700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f48bc005010 tx=0x7f48bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:15.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48bc009320 con 0x7f48cc0f5410 2024-02-13T03:32:15.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc105dd0 con 0x7f48cc0f5410 2024-02-13T03:32:15.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48bc013380 con 0x7f48cc0f5410 2024-02-13T03:32:15.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.577+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48bc014e50 con 0x7f48cc0f5410 2024-02-13T03:32:15.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.578+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc106260 con 0x7f48cc0f5410 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.578+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48bc014420 con 0x7f48cc0f5410 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.578+0000 7f48c1ffb700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 0x7f48b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.578+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48bc040f70 con 0x7f48cc0f5410 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.578+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc045ad0 con 0x7f48cc0f5410 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.579+0000 7f48d0d20700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 0x7f48b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:15.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.579+0000 7f48d0d20700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 0x7f48b80323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f48c80096a0 tx=0x7f48c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:15.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.585+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48bc0095f0 con 0x7f48cc0f5410 2024-02-13T03:32:15.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.788+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48cc0fb090 con 0x7f48cc0f5410 2024-02-13T03:32:15.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.789+0000 7f48c1ffb700 1 -- 172.21.15.112:0/3891037265 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48bc01db70 con 0x7f48cc0f5410 2024-02-13T03:32:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:15.790 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 msgr2=0x7f48b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 0x7f48b80323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f48c80096a0 tx=0x7f48c8006b40).stop 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 msgr2=0x7f48cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f48bc005010 tx=0x7f48bc00b040).stop 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 shutdown_connections 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f48b802ff10 0x7f48b80323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 --2- 172.21.15.112:0/3891037265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f48cc0f5410 0x7f48cc1056e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:15.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 >> 172.21.15.112:0/3891037265 conn(0x7f48cc0f0b70 msgr2=0x7f48cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:15.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 shutdown_connections 2024-02-13T03:32:15.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:15.790+0000 7f48d3785700 1 -- 172.21.15.112:0/3891037265 wait complete. 2024-02-13T03:32:15.792 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:16.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:14.771516+0000 mgr.smithi080.vyzlfg (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:16.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:15.790110+0000 mon.smithi080 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.112:0/3891037265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:17.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:17.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:18.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:16.771718+0000 mgr.smithi080.vyzlfg (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:19.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 -- 172.21.15.112:0/2764233871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 msgr2=0x7fe7440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:19.451 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 --2- 172.21.15.112:0/2764233871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7440f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe738009fb0 tx=0x7fe738009b30).stop 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 -- 172.21.15.112:0/2764233871 shutdown_connections 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 --2- 172.21.15.112:0/2764233871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7440f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 -- 172.21.15.112:0/2764233871 >> 172.21.15.112:0/2764233871 conn(0x7fe7440f0b70 msgr2=0x7fe7440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 -- 172.21.15.112:0/2764233871 shutdown_connections 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.420+0000 7fe749faf700 1 -- 172.21.15.112:0/2764233871 wait complete. 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe749faf700 1 Processor -- start 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe749faf700 1 -- start start 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe749faf700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe749faf700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744105bd0 con 0x7fe7440f5410 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe7437fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe7437fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:46910/0 (socket says 172.21.15.112:46910) 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe7437fe700 1 -- 172.21.15.112:0/302589918 learned_addr learned my addr 172.21.15.112:0/302589918 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe7437fe700 1 -- 172.21.15.112:0/302589918 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738005040 con 0x7fe7440f5410 2024-02-13T03:32:19.452 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.421+0000 7fe7437fe700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fe738005010 tx=0x7fe73800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe738009320 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe738013380 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe744105dd0 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe738014e50 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe744106260 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7380263a0 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe72bfff700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.422+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe744045ad0 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.423+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe738040fa0 con 0x7fe7440f5410 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.423+0000 7fe742ffd700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.423+0000 7fe742ffd700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 0x7fe72c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe7340096a0 tx=0x7fe734006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:19.453 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.427+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe738012510 con 0x7fe7440f5410 2024-02-13T03:32:19.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.634+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe744107110 con 0x7fe7440f5410 2024-02-13T03:32:19.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.635+0000 7fe72bfff700 1 -- 172.21.15.112:0/302589918 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe73800f340 con 0x7fe7440f5410 2024-02-13T03:32:19.636 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:19.636 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:19.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 msgr2=0x7fe72c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:19.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 0x7fe72c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe7340096a0 tx=0x7fe734006b40).stop 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 msgr2=0x7fe7441056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fe738005010 tx=0x7fe73800b040).stop 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 shutdown_connections 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 --2- 172.21.15.112:0/302589918 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe7440f5410 0x7fe7441056e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 >> 172.21.15.112:0/302589918 conn(0x7fe7440f0b70 msgr2=0x7fe7440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 shutdown_connections 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:19.637+0000 7fe749faf700 1 -- 172.21.15.112:0/302589918 wait complete. 2024-02-13T03:32:19.638 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:20.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:18.771951+0000 mgr.smithi080.vyzlfg (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:20.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:19.636233+0000 mon.smithi080 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.112:0/302589918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:21.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:21.272 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:22.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:20.772210+0000 mgr.smithi080.vyzlfg (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:23.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- 172.21.15.112:0/3336450064 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 msgr2=0x7f71340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 --2- 172.21.15.112:0/3336450064 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71340f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f7130009fb0 tx=0x7f7130009b30).stop 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- 172.21.15.112:0/3336450064 shutdown_connections 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 --2- 172.21.15.112:0/3336450064 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71340f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- 172.21.15.112:0/3336450064 >> 172.21.15.112:0/3336450064 conn(0x7f71340f09a0 msgr2=0x7f71340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- 172.21.15.112:0/3336450064 shutdown_connections 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- 172.21.15.112:0/3336450064 wait complete. 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 Processor -- start 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- start start 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.110+0000 7f713a29d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7134106a90 con 0x7f71340f51a0 2024-02-13T03:32:23.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713929b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713929b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:39370/0 (socket says 172.21.15.112:39370) 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713929b700 1 -- 172.21.15.112:0/3283707299 learned_addr learned my addr 172.21.15.112:0/3283707299 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713929b700 1 -- 172.21.15.112:0/3283707299 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130005040 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713929b700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f71300089f0 tx=0x7f713000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130009320 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7134106c90 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7130013380 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130014e20 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.111+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7134107120 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.112+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134045ad0 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.112+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7130014420 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.112+0000 7f7129ffb700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 0x7f71200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.112+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7130040ef0 con 0x7f71340f51a0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.113+0000 7f7138a9a700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 0x7f71200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:23.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.113+0000 7f7138a9a700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 0x7f71200323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f71240096a0 tx=0x7f7124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:23.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.117+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7130014790 con 0x7f71340f51a0 2024-02-13T03:32:23.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.320+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7134107f70 con 0x7f71340f51a0 2024-02-13T03:32:23.322 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.321+0000 7f7129ffb700 1 -- 172.21.15.112:0/3283707299 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f713001d7f0 con 0x7f71340f51a0 2024-02-13T03:32:23.322 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:23.322 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 msgr2=0x7f71200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 0x7f71200323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f71240096a0 tx=0x7f7124006b40).stop 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 msgr2=0x7f71341065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f71300089f0 tx=0x7f713000b040).stop 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 shutdown_connections 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f712002ff10 0x7f71200323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 --2- 172.21.15.112:0/3283707299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f71340f51a0 0x7f71341065a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 >> 172.21.15.112:0/3283707299 conn(0x7f71340f09a0 msgr2=0x7f71340f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 shutdown_connections 2024-02-13T03:32:23.324 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:23.323+0000 7f713a29d700 1 -- 172.21.15.112:0/3283707299 wait complete. 2024-02-13T03:32:23.325 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:23.540 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:23 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:23.322629+0000 mon.smithi080 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.112:0/3283707299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:24.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:22.772416+0000 mgr.smithi080.vyzlfg (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:25.038 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:25.039 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:26.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:24.772789+0000 mgr.smithi080.vyzlfg (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:26.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:25.267893+0000 mon.smithi080 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:32:26.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:32:25.269189+0000 mgr.smithi080.vyzlfg (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi080 on smithi080 2024-02-13T03:32:26.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.894+0000 7f14a2bc3700 1 -- 172.21.15.112:0/3990217876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 msgr2=0x7f149c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:26.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.894+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/3990217876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c0f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f1490009fb0 tx=0x7f1490009b30).stop 2024-02-13T03:32:26.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.894+0000 7f14a2bc3700 1 -- 172.21.15.112:0/3990217876 shutdown_connections 2024-02-13T03:32:26.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.894+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/3990217876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c0f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:26.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.894+0000 7f14a2bc3700 1 -- 172.21.15.112:0/3990217876 >> 172.21.15.112:0/3990217876 conn(0x7f149c0f1c70 msgr2=0x7f149c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:26.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.895+0000 7f14a2bc3700 1 -- 172.21.15.112:0/3990217876 shutdown_connections 2024-02-13T03:32:26.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.895+0000 7f14a2bc3700 1 -- 172.21.15.112:0/3990217876 wait complete. 2024-02-13T03:32:26.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.895+0000 7f14a2bc3700 1 Processor -- start 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.895+0000 7f14a2bc3700 1 -- start start 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a2bc3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a2bc3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c106cd0 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a095f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a095f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:39398/0 (socket says 172.21.15.112:39398) 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a095f700 1 -- 172.21.15.112:0/1575557186 learned_addr learned my addr 172.21.15.112:0/1575557186 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a095f700 1 -- 172.21.15.112:0/1575557186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490005040 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.896+0000 7f14a095f700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f1490005010 tx=0x7f149000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1490009320 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c106ed0 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1490013380 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1490014e50 con 0x7f149c0f6510 2024-02-13T03:32:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c107360 con 0x7f149c0f6510 2024-02-13T03:32:26.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1490014420 con 0x7f149c0f6510 2024-02-13T03:32:26.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c045ad0 con 0x7f149c0f6510 2024-02-13T03:32:26.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.897+0000 7f14997fa700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 0x7f148c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:26.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.898+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1490040f40 con 0x7f149c0f6510 2024-02-13T03:32:26.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.898+0000 7f149bfff700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 0x7f148c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:26.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.898+0000 7f149bfff700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 0x7f148c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f14880096a0 tx=0x7f1488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:26.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:26.902+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1490012330 con 0x7f149c0f6510 2024-02-13T03:32:27.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.108+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f149c108210 con 0x7f149c0f6510 2024-02-13T03:32:27.109 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.109+0000 7f14997fa700 1 -- 172.21.15.112:0/1575557186 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f149001dd50 con 0x7f149c0f6510 2024-02-13T03:32:27.110 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:27.110 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:27.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 msgr2=0x7f148c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 0x7f148c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f14880096a0 tx=0x7f1488006b40).stop 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 msgr2=0x7f149c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f1490005010 tx=0x7f149000b040).stop 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 shutdown_connections 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f148c02ff10 0x7f148c0323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 --2- 172.21.15.112:0/1575557186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 >> 172.21.15.112:0/1575557186 conn(0x7f149c0f1c70 msgr2=0x7f149c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.110+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 shutdown_connections 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:27.111+0000 7f14a2bc3700 1 -- 172.21.15.112:0/1575557186 wait complete. 2024-02-13T03:32:27.112 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:27.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:27.110320+0000 mon.smithi080 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.112:0/1575557186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:28.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:26.773219+0000 mgr.smithi080.vyzlfg (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:28.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:28.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:30.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:28.773533+0000 mgr.smithi080.vyzlfg (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:30.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 -- 172.21.15.112:0/415451225 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 msgr2=0x7fdaa80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 --2- 172.21.15.112:0/415451225 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa80f6930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdaa4009fb0 tx=0x7fdaa4009b30).stop 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 -- 172.21.15.112:0/415451225 shutdown_connections 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 --2- 172.21.15.112:0/415451225 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa80f6930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 -- 172.21.15.112:0/415451225 >> 172.21.15.112:0/415451225 conn(0x7fdaa80f1c70 msgr2=0x7fdaa80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 -- 172.21.15.112:0/415451225 shutdown_connections 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.930+0000 7fdab0497700 1 -- 172.21.15.112:0/415451225 wait complete. 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdab0497700 1 Processor -- start 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdab0497700 1 -- start start 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdab0497700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdab0497700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8106cd0 con 0x7fdaa80f6510 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdaae233700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdaae233700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:39416/0 (socket says 172.21.15.112:39416) 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.931+0000 7fdaae233700 1 -- 172.21.15.112:0/4079310677 learned_addr learned my addr 172.21.15.112:0/4079310677 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:30.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fdaae233700 1 -- 172.21.15.112:0/4079310677 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4005040 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fdaae233700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdaa40089f0 tx=0x7fdaa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa4009320 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8106ed0 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa4013380 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa401ed80 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.932+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa8107360 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.933+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaa401e420 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.933+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa8045ad0 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.933+0000 7fda9effd700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.933+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaa4041150 con 0x7fdaa80f6510 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.933+0000 7fdaada32700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.934+0000 7fdaada32700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 0x7fda94032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fda980096a0 tx=0x7fda98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:30.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:30.939+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaa40095f0 con 0x7fdaa80f6510 2024-02-13T03:32:31.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.144+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdaa8108210 con 0x7fdaa80f6510 2024-02-13T03:32:31.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.145+0000 7fda9effd700 1 -- 172.21.15.112:0/4079310677 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaa4004490 con 0x7fdaa80f6510 2024-02-13T03:32:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:31.146 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:31.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 msgr2=0x7fda94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:31.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 0x7fda94032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fda980096a0 tx=0x7fda98006b40).stop 2024-02-13T03:32:31.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 msgr2=0x7fdaa81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:31.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdaa40089f0 tx=0x7fdaa400b040).stop 2024-02-13T03:32:31.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 shutdown_connections 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fda9402ff60 0x7fda94032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 --2- 172.21.15.112:0/4079310677 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdaa80f6510 0x7fdaa81067e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 >> 172.21.15.112:0/4079310677 conn(0x7fdaa80f1c70 msgr2=0x7fdaa80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 shutdown_connections 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:31.146+0000 7fdab0497700 1 -- 172.21.15.112:0/4079310677 wait complete. 2024-02-13T03:32:31.148 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:31.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:31 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:31.145973+0000 mon.smithi080 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.112:0/4079310677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:32.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:30.773839+0000 mgr.smithi080.vyzlfg (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:32.817 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:32.818 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:34.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:32.774200+0000 mgr.smithi080.vyzlfg (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3691266669 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 msgr2=0x7f7e040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3691266669 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f90f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f7df8009fb0 tx=0x7f7df8009b30).stop 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3691266669 shutdown_connections 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3691266669 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f90f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3691266669 >> 172.21.15.112:0/3691266669 conn(0x7f7e040f0b70 msgr2=0x7f7e040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3691266669 shutdown_connections 2024-02-13T03:32:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.692+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3691266669 wait complete. 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e0aa8b700 1 Processor -- start 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e0aa8b700 1 -- start start 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e0aa8b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e0aa8b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040f4ff0 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e08827700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e08827700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:40724/0 (socket says 172.21.15.112:40724) 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.693+0000 7f7e08827700 1 -- 172.21.15.112:0/3062040083 learned_addr learned my addr 172.21.15.112:0/3062040083 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e08827700 1 -- 172.21.15.112:0/3062040083 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8005040 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e08827700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7df8009830 tx=0x7f7df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df8009320 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040f5190 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7df8013380 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df8014420 con 0x7f7e04060280 2024-02-13T03:32:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.694+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e040f5620 con 0x7f7e04060280 2024-02-13T03:32:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7df801e420 con 0x7f7e04060280 2024-02-13T03:32:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e04059100 con 0x7f7e04060280 2024-02-13T03:32:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e017fa700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 0x7f7df4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7df8040df0 con 0x7f7e04060280 2024-02-13T03:32:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e03fff700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 0x7f7df4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.695+0000 7f7e03fff700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 0x7f7df4032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7df00096a0 tx=0x7f7df0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:34.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.699+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7df801cb10 con 0x7f7e04060280 2024-02-13T03:32:34.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.906+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e04045ad0 con 0x7f7e04060280 2024-02-13T03:32:34.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.906+0000 7f7e017fa700 1 -- 172.21.15.112:0/3062040083 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7df802c2d0 con 0x7f7e04060280 2024-02-13T03:32:34.906 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:34.906 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 msgr2=0x7f7df4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 0x7f7df4032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7df00096a0 tx=0x7f7df0006b40).stop 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 msgr2=0x7f7e040f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7df8009830 tx=0x7f7df800b040).stop 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 shutdown_connections 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7df402ff60 0x7f7df4032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 --2- 172.21.15.112:0/3062040083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7e04060280 0x7f7e040f6b70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 >> 172.21.15.112:0/3062040083 conn(0x7f7e040f0b70 msgr2=0x7f7e040f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 shutdown_connections 2024-02-13T03:32:34.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:34.908+0000 7f7e0aa8b700 1 -- 172.21.15.112:0/3062040083 wait complete. 2024-02-13T03:32:34.909 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:35.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:35 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:34.906925+0000 mon.smithi080 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.112:0/3062040083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:36.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:36.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:36.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:34.774533+0000 mgr.smithi080.vyzlfg (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 -- 172.21.15.112:0/2901198564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 msgr2=0x7f50180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 --2- 172.21.15.112:0/2901198564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50180f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5000009fb0 tx=0x7f5000009b30).stop 2024-02-13T03:32:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 -- 172.21.15.112:0/2901198564 shutdown_connections 2024-02-13T03:32:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 --2- 172.21.15.112:0/2901198564 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50180f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 -- 172.21.15.112:0/2901198564 >> 172.21.15.112:0/2901198564 conn(0x7f50180f1c70 msgr2=0x7f50180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 -- 172.21.15.112:0/2901198564 shutdown_connections 2024-02-13T03:32:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.618+0000 7f501d459700 1 -- 172.21.15.112:0/2901198564 wait complete. 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f501d459700 1 Processor -- start 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f501d459700 1 -- start start 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f501d459700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f501d459700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018106cd0 con 0x7f50180f6510 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.619+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:40740/0 (socket says 172.21.15.112:40740) 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f5016ffd700 1 -- 172.21.15.112:0/315539451 learned_addr learned my addr 172.21.15.112:0/315539451 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f5016ffd700 1 -- 172.21.15.112:0/315539451 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5000005040 con 0x7f50180f6510 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f5016ffd700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5000005010 tx=0x7f500000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000009320 con 0x7f50180f6510 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5018106ed0 con 0x7f50180f6510 2024-02-13T03:32:38.621 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5000013380 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000014e50 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.620+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5018107360 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.621+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5000014420 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.621+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5018045ad0 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.621+0000 7f500f7fe700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.621+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5000040e30 con 0x7f50180f6510 2024-02-13T03:32:38.622 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.621+0000 7f50167fc700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:38.623 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.622+0000 7f50167fc700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:38.625 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.626+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50000095f0 con 0x7f50180f6510 2024-02-13T03:32:38.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:36.774908+0000 mgr.smithi080.vyzlfg (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:38.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.829+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5018108210 con 0x7f50180f6510 2024-02-13T03:32:38.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.829+0000 7f500f7fe700 1 -- 172.21.15.112:0/315539451 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f500001db70 con 0x7f50180f6510 2024-02-13T03:32:38.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:38.829 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 msgr2=0x7f50040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).stop 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 msgr2=0x7f50181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f5000005010 tx=0x7f500000b040).stop 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 shutdown_connections 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f500402ff10 0x7f50040323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 --2- 172.21.15.112:0/315539451 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50180f6510 0x7f50181067e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 >> 172.21.15.112:0/315539451 conn(0x7f50180f1c70 msgr2=0x7f50180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 shutdown_connections 2024-02-13T03:32:38.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:38.831+0000 7f501d459700 1 -- 172.21.15.112:0/315539451 wait complete. 2024-02-13T03:32:38.832 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:39.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:38.829647+0000 mon.smithi080 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.112:0/315539451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:40.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:40.514 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:40.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:38.775277+0000 mgr.smithi080.vyzlfg (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:42.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 -- 172.21.15.112:0/4127282379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb8600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:42.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 --2- 172.21.15.112:0/4127282379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fb848009fb0 tx=0x7fb848009b30).stop 2024-02-13T03:32:42.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 -- 172.21.15.112:0/4127282379 shutdown_connections 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 --2- 172.21.15.112:0/4127282379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 -- 172.21.15.112:0/4127282379 >> 172.21.15.112:0/4127282379 conn(0x7fb8600f0b70 msgr2=0x7fb8600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 -- 172.21.15.112:0/4127282379 shutdown_connections 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.430+0000 7fb865b1d700 1 -- 172.21.15.112:0/4127282379 wait complete. 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb865b1d700 1 Processor -- start 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb865b1d700 1 -- start start 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb865b1d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb865b1d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860106cc0 con 0x7fb8600f5410 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb85f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb85f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50828/0 (socket says 172.21.15.112:50828) 2024-02-13T03:32:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.431+0000 7fb85f7fe700 1 -- 172.21.15.112:0/2336992700 learned_addr learned my addr 172.21.15.112:0/2336992700 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb85f7fe700 1 -- 172.21.15.112:0/2336992700 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848005040 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb85f7fe700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb848005010 tx=0x7fb84800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb848009320 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb860106ec0 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb848013380 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb848014e50 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.432+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb860107350 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8480095f0 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860045ad0 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb857fff700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 0x7fb84c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb848040e90 con 0x7fb8600f5410 2024-02-13T03:32:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb85effd700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 0x7fb84c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:42.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.433+0000 7fb85effd700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 0x7fb84c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:42.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.437+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb848014420 con 0x7fb8600f5410 2024-02-13T03:32:42.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.643+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb860108200 con 0x7fb8600f5410 2024-02-13T03:32:42.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.644+0000 7fb857fff700 1 -- 172.21.15.112:0/2336992700 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb848010230 con 0x7fb8600f5410 2024-02-13T03:32:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:42.645 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 msgr2=0x7fb84c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 0x7fb84c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).stop 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb8601067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb848005010 tx=0x7fb84800b040).stop 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 shutdown_connections 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb84c02ff10 0x7fb84c0323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 --2- 172.21.15.112:0/2336992700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8600f5410 0x7fb8601067d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.645+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 >> 172.21.15.112:0/2336992700 conn(0x7fb8600f0b70 msgr2=0x7fb86005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.646+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 shutdown_connections 2024-02-13T03:32:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:42.646+0000 7fb865b1d700 1 -- 172.21.15.112:0/2336992700 wait complete. 2024-02-13T03:32:42.648 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:42.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:40.775640+0000 mgr.smithi080.vyzlfg (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:43.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:42.645271+0000 mon.smithi080 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.112:0/2336992700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:44.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:44.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:42.775924+0000 mgr.smithi080.vyzlfg (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:46.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- 172.21.15.112:0/1314293044 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 msgr2=0x7efcf00f4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:46.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1314293044 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf00f4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7efce4009fb0 tx=0x7efce4009b30).stop 2024-02-13T03:32:46.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- 172.21.15.112:0/1314293044 shutdown_connections 2024-02-13T03:32:46.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1314293044 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf00f4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- 172.21.15.112:0/1314293044 >> 172.21.15.112:0/1314293044 conn(0x7efcf00f02e0 msgr2=0x7efcf00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- 172.21.15.112:0/1314293044 shutdown_connections 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- 172.21.15.112:0/1314293044 wait complete. 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 Processor -- start 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.309+0000 7efcf6aba700 1 -- start start 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf6aba700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf6aba700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0106cc0 con 0x7efcf00f48a0 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf4856700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf4856700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50834/0 (socket says 172.21.15.112:50834) 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf4856700 1 -- 172.21.15.112:0/1296240157 learned_addr learned my addr 172.21.15.112:0/1296240157 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.310+0000 7efcf4856700 1 -- 172.21.15.112:0/1296240157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce4005040 con 0x7efcf00f48a0 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efcf4856700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7efce4005010 tx=0x7efce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce4009320 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf0106ec0 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efce4013380 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce4014e50 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.311+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf0107350 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efce4014420 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efced7fa700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 0x7efce00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efce4040e30 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0045ad0 con 0x7efcf00f48a0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efceffff700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 0x7efce00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.312+0000 7efceffff700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 0x7efce00323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efcdc0096a0 tx=0x7efcdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.316+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efce40095f0 con 0x7efcf00f48a0 2024-02-13T03:32:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.521+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efcf0108200 con 0x7efcf00f48a0 2024-02-13T03:32:46.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.521+0000 7efced7fa700 1 -- 172.21.15.112:0/1296240157 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efce402b020 con 0x7efcf00f48a0 2024-02-13T03:32:46.522 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:46.522 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 msgr2=0x7efce00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 0x7efce00323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efcdc0096a0 tx=0x7efcdc006b40).stop 2024-02-13T03:32:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 msgr2=0x7efcf01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7efce4005010 tx=0x7efce400b040).stop 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 shutdown_connections 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efce002ff10 0x7efce00323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 --2- 172.21.15.112:0/1296240157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcf00f48a0 0x7efcf01067d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 >> 172.21.15.112:0/1296240157 conn(0x7efcf00f02e0 msgr2=0x7efcf00f0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 shutdown_connections 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:46.523+0000 7efcf6aba700 1 -- 172.21.15.112:0/1296240157 wait complete. 2024-02-13T03:32:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:44.776257+0000 mgr.smithi080.vyzlfg (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:47.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:46.522776+0000 mon.smithi080 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.112:0/1296240157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:48.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:48.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:46.776611+0000 mgr.smithi080.vyzlfg (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- 172.21.15.112:0/668551259 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 msgr2=0x7ff43c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 --2- 172.21.15.112:0/668551259 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c0f90f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7ff42c009fb0 tx=0x7ff42c009b30).stop 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- 172.21.15.112:0/668551259 shutdown_connections 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 --2- 172.21.15.112:0/668551259 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c0f90f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- 172.21.15.112:0/668551259 >> 172.21.15.112:0/668551259 conn(0x7ff43c0f0b70 msgr2=0x7ff43c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- 172.21.15.112:0/668551259 shutdown_connections 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- 172.21.15.112:0/668551259 wait complete. 2024-02-13T03:32:50.284 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 Processor -- start 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.273+0000 7ff4437c4700 1 -- start start 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff4437c4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff4437c4700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c106c70 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff441560700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff441560700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50852/0 (socket says 172.21.15.112:50852) 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff441560700 1 -- 172.21.15.112:0/1513943304 learned_addr learned my addr 172.21.15.112:0/1513943304 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff441560700 1 -- 172.21.15.112:0/1513943304 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c005040 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff441560700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff42c0089f0 tx=0x7ff42c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c009320 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.274+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c106e70 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.275+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c107300 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.275+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff42c013380 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.275+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff42c014e40 con 0x7ff43c060280 2024-02-13T03:32:50.285 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.275+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c045ad0 con 0x7ff43c060280 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.275+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff42c014420 con 0x7ff43c060280 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.276+0000 7ff431ffb700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 0x7ff4280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.276+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff42c041060 con 0x7ff43c060280 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.276+0000 7ff440d5f700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 0x7ff4280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.276+0000 7ff440d5f700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 0x7ff4280323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff4380096a0 tx=0x7ff438006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:50.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.280+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff42c06d2b0 con 0x7ff43c060280 2024-02-13T03:32:50.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.489+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff43c1081b0 con 0x7ff43c060280 2024-02-13T03:32:50.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.490+0000 7ff431ffb700 1 -- 172.21.15.112:0/1513943304 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff42c012bb0 con 0x7ff43c060280 2024-02-13T03:32:50.490 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:50.490 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 msgr2=0x7ff4280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 0x7ff4280323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff4380096a0 tx=0x7ff438006b40).stop 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 msgr2=0x7ff43c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff42c0089f0 tx=0x7ff42c00b040).stop 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 shutdown_connections 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff42802ff10 0x7ff4280323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 --2- 172.21.15.112:0/1513943304 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff43c060280 0x7ff43c106780 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:50.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 >> 172.21.15.112:0/1513943304 conn(0x7ff43c0f0b70 msgr2=0x7ff43c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:50.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 shutdown_connections 2024-02-13T03:32:50.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:50.491+0000 7ff4437c4700 1 -- 172.21.15.112:0/1513943304 wait complete. 2024-02-13T03:32:50.493 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:48.776965+0000 mgr.smithi080.vyzlfg (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:51.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:50.490880+0000 mon.smithi080 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.112:0/1513943304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:52.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:52.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:50.777241+0000 mgr.smithi080.vyzlfg (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- 172.21.15.112:0/3909818209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 msgr2=0x7efc780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/3909818209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc780f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7efc6c009fb0 tx=0x7efc6c009b30).stop 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- 172.21.15.112:0/3909818209 shutdown_connections 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/3909818209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc780f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- 172.21.15.112:0/3909818209 >> 172.21.15.112:0/3909818209 conn(0x7efc780f1c70 msgr2=0x7efc780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:54.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- 172.21.15.112:0/3909818209 shutdown_connections 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- 172.21.15.112:0/3909818209 wait complete. 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 Processor -- start 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.145+0000 7efc7ef9e700 1 -- start start 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7ef9e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7ef9e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78106cd0 con 0x7efc780f6510 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7cd3a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7cd3a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33198/0 (socket says 172.21.15.112:33198) 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7cd3a700 1 -- 172.21.15.112:0/576893124 learned_addr learned my addr 172.21.15.112:0/576893124 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7cd3a700 1 -- 172.21.15.112:0/576893124 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c005040 con 0x7efc780f6510 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7cd3a700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7efc6c005010 tx=0x7efc6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6c009320 con 0x7efc780f6510 2024-02-13T03:32:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc6c013380 con 0x7efc780f6510 2024-02-13T03:32:54.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc78106ed0 con 0x7efc780f6510 2024-02-13T03:32:54.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6c014e50 con 0x7efc780f6510 2024-02-13T03:32:54.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.146+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc78107360 con 0x7efc780f6510 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.147+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc6c0095f0 con 0x7efc780f6510 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.147+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc78045ad0 con 0x7efc780f6510 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.147+0000 7efc757fa700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 0x7efc680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.147+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc6c040f10 con 0x7efc780f6510 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.148+0000 7efc77fff700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 0x7efc680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:54.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.148+0000 7efc77fff700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 0x7efc680323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efc640096a0 tx=0x7efc64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:54.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.152+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc6c014420 con 0x7efc780f6510 2024-02-13T03:32:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.357+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc78108210 con 0x7efc780f6510 2024-02-13T03:32:54.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.358+0000 7efc757fa700 1 -- 172.21.15.112:0/576893124 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc6c010230 con 0x7efc780f6510 2024-02-13T03:32:54.359 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:54.359 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 msgr2=0x7efc680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:54.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 0x7efc680323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efc640096a0 tx=0x7efc64006b40).stop 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 msgr2=0x7efc781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7efc6c005010 tx=0x7efc6c00b040).stop 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 shutdown_connections 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efc6802ff10 0x7efc680323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 --2- 172.21.15.112:0/576893124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc780f6510 0x7efc781067e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.359+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 >> 172.21.15.112:0/576893124 conn(0x7efc780f1c70 msgr2=0x7efc780f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.360+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 shutdown_connections 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:54.360+0000 7efc7ef9e700 1 -- 172.21.15.112:0/576893124 wait complete. 2024-02-13T03:32:54.361 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:52.777576+0000 mgr.smithi080.vyzlfg (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:54.359265+0000 mon.smithi080 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.112:0/576893124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:56.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:56.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:32:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:54.777940+0000 mgr.smithi080.vyzlfg (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:56.859355+0000 mon.smithi080 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:32:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:56.866774+0000 mon.smithi080 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:32:57.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.998+0000 7f3c20156700 1 -- 172.21.15.112:0/411422124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 msgr2=0x7f3c180f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:57.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.998+0000 7f3c20156700 1 --2- 172.21.15.112:0/411422124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f9150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3c0c009fb0 tx=0x7f3c0c009b30).stop 2024-02-13T03:32:57.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- 172.21.15.112:0/411422124 shutdown_connections 2024-02-13T03:32:57.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 --2- 172.21.15.112:0/411422124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f9150 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:57.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- 172.21.15.112:0/411422124 >> 172.21.15.112:0/411422124 conn(0x7f3c180f0b70 msgr2=0x7f3c180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:58.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- 172.21.15.112:0/411422124 shutdown_connections 2024-02-13T03:32:58.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- 172.21.15.112:0/411422124 wait complete. 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 Processor -- start 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- start start 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:57.999+0000 7f3c20156700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180f5050 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c1def2700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c1def2700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:33216/0 (socket says 172.21.15.112:33216) 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c1def2700 1 -- 172.21.15.112:0/1686114628 learned_addr learned my addr 172.21.15.112:0/1686114628 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c1def2700 1 -- 172.21.15.112:0/1686114628 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c005040 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c1def2700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3c0c009830 tx=0x7f3c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0c009320 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c180f51f0 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c0c013380 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0c014420 con 0x7f3c18060220 2024-02-13T03:32:58.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.000+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c180f5680 con 0x7f3c18060220 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.001+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c0c01e420 con 0x7f3c18060220 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.001+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c18059100 con 0x7f3c18060220 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.001+0000 7f3c0a7fc700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.001+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c0c040e20 con 0x7f3c18060220 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.001+0000 7f3c1d6f1700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:32:58.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.002+0000 7f3c1d6f1700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 0x7f3c04032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c140096a0 tx=0x7f3c14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:32:58.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.006+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c0c01c970 con 0x7f3c18060220 2024-02-13T03:32:58.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.210+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c18045ad0 con 0x7f3c18060220 2024-02-13T03:32:58.212 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.211+0000 7f3c0a7fc700 1 -- 172.21.15.112:0/1686114628 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c0c02c2d0 con 0x7f3c18060220 2024-02-13T03:32:58.212 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:32:58.212 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:32:58.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 msgr2=0x7f3c04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 0x7f3c04032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c140096a0 tx=0x7f3c14006b40).stop 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 msgr2=0x7f3c180f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3c0c009830 tx=0x7f3c0c00b040).stop 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 shutdown_connections 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 --2- 172.21.15.112:0/1686114628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c18060220 0x7f3c180f6bd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 >> 172.21.15.112:0/1686114628 conn(0x7f3c180f0b70 msgr2=0x7f3c180f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 shutdown_connections 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:32:58.213+0000 7f3c20156700 1 -- 172.21.15.112:0/1686114628 wait complete. 2024-02-13T03:32:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:32:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:56.778299+0000 mgr.smithi080.vyzlfg (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:32:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:32:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:32:58.212430+0000 mon.smithi080 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.112:0/1686114628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:32:59.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:32:59.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:32:58.778571+0000 mgr.smithi080.vyzlfg (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:01.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.766+0000 7f30fdec3700 1 -- 172.21.15.112:0/801331841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 msgr2=0x7f30f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:01.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.766+0000 7f30fdec3700 1 --2- 172.21.15.112:0/801331841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f80f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f30ec009fb0 tx=0x7f30ec009b30).stop 2024-02-13T03:33:01.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 -- 172.21.15.112:0/801331841 shutdown_connections 2024-02-13T03:33:01.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 --2- 172.21.15.112:0/801331841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f80f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:01.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 -- 172.21.15.112:0/801331841 >> 172.21.15.112:0/801331841 conn(0x7f30f80f09a0 msgr2=0x7f30f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:01.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 -- 172.21.15.112:0/801331841 shutdown_connections 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 -- 172.21.15.112:0/801331841 wait complete. 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 Processor -- start 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.767+0000 7f30fdec3700 1 -- start start 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fdec3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fdec3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81059c0 con 0x7f30f80f51a0 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fcec1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fcec1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47162/0 (socket says 172.21.15.112:47162) 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fcec1700 1 -- 172.21.15.112:0/3467556626 learned_addr learned my addr 172.21.15.112:0/3467556626 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.768+0000 7f30fcec1700 1 -- 172.21.15.112:0/3467556626 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec005040 con 0x7f30f80f51a0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30fcec1700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f30ec0089f0 tx=0x7f30ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ec009320 con 0x7f30f80f51a0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f8105bc0 con 0x7f30f80f51a0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30ec013380 con 0x7f30f80f51a0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f8106050 con 0x7f30f80f51a0 2024-02-13T03:33:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.769+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ec014e20 con 0x7f30f80f51a0 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30ec014420 con 0x7f30f80f51a0 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f8045ad0 con 0x7f30f80f51a0 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30f57fa700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 0x7f30e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30ec040f70 con 0x7f30f80f51a0 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30f7fff700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 0x7f30e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.770+0000 7f30f7fff700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 0x7f30e80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f30e40096a0 tx=0x7f30e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.774+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30ec014790 con 0x7f30f80f51a0 2024-02-13T03:33:01.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.982+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30f8106ea0 con 0x7f30f80f51a0 2024-02-13T03:33:01.983 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.983+0000 7f30f57fa700 1 -- 172.21.15.112:0/3467556626 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f30ec010230 con 0x7f30f80f51a0 2024-02-13T03:33:01.984 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:01.984 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:01.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.984+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 msgr2=0x7f30e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:01.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.984+0000 7f30fdec3700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 0x7f30e80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f30e40096a0 tx=0x7f30e4006b40).stop 2024-02-13T03:33:01.985 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.984+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 msgr2=0x7f30f81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.984+0000 7f30fdec3700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f30ec0089f0 tx=0x7f30ec00b040).stop 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 shutdown_connections 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f30e802ff10 0x7f30e80323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 --2- 172.21.15.112:0/3467556626 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30f80f51a0 0x7f30f81054d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 >> 172.21.15.112:0/3467556626 conn(0x7f30f80f09a0 msgr2=0x7f30f80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 shutdown_connections 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:01.985+0000 7f30fdec3700 1 -- 172.21.15.112:0/3467556626 wait complete. 2024-02-13T03:33:01.986 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:02.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:00.778911+0000 mgr.smithi080.vyzlfg (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:02.691 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:01.984372+0000 mon.smithi080 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.112:0/3467556626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:03.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:03.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:02.779267+0000 mgr.smithi080.vyzlfg (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.566+0000 7fb9ab225700 1 -- 172.21.15.112:0/3747283844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 msgr2=0x7fb9a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.566+0000 7fb9ab225700 1 --2- 172.21.15.112:0/3747283844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a40f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb998009fb0 tx=0x7fb998009b30).stop 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 -- 172.21.15.112:0/3747283844 shutdown_connections 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 --2- 172.21.15.112:0/3747283844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a40f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 -- 172.21.15.112:0/3747283844 >> 172.21.15.112:0/3747283844 conn(0x7fb9a40f0b70 msgr2=0x7fb9a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 -- 172.21.15.112:0/3747283844 shutdown_connections 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 -- 172.21.15.112:0/3747283844 wait complete. 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 Processor -- start 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.567+0000 7fb9ab225700 1 -- start start 2024-02-13T03:33:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9ab225700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9ab225700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4105bd0 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a8fc1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a8fc1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47174/0 (socket says 172.21.15.112:47174) 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a8fc1700 1 -- 172.21.15.112:0/4225781392 learned_addr learned my addr 172.21.15.112:0/4225781392 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a8fc1700 1 -- 172.21.15.112:0/4225781392 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb998005040 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a8fc1700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb998005010 tx=0x7fb99800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998009320 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a4105dd0 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.568+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb998013380 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998014e50 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a4106260 con 0x7fb9a40f5410 2024-02-13T03:33:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9980263a0 con 0x7fb9a40f5410 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9a17fa700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 0x7fb9940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9980410e0 con 0x7fb9a40f5410 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.569+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a4045ad0 con 0x7fb9a40f5410 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.570+0000 7fb9a3fff700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 0x7fb9940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.570+0000 7fb9a3fff700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 0x7fb9940323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb9900096a0 tx=0x7fb990006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.574+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9980267a0 con 0x7fb9a40f5410 2024-02-13T03:33:05.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.778+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9a40fb090 con 0x7fb9a40f5410 2024-02-13T03:33:05.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.779+0000 7fb9a17fa700 1 -- 172.21.15.112:0/4225781392 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb99801cb10 con 0x7fb9a40f5410 2024-02-13T03:33:05.780 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:05.780 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 msgr2=0x7fb9940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 0x7fb9940323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb9900096a0 tx=0x7fb990006b40).stop 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 msgr2=0x7fb9a41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb998005010 tx=0x7fb99800b040).stop 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 shutdown_connections 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb99402ff10 0x7fb9940323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:05.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 --2- 172.21.15.112:0/4225781392 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb9a40f5410 0x7fb9a41056e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:05.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 >> 172.21.15.112:0/4225781392 conn(0x7fb9a40f0b70 msgr2=0x7fb9a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:05.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 shutdown_connections 2024-02-13T03:33:05.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:05.780+0000 7fb9ab225700 1 -- 172.21.15.112:0/4225781392 wait complete. 2024-02-13T03:33:05.782 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:04.779569+0000 mgr.smithi080.vyzlfg (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:05.780050+0000 mon.smithi080 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.112:0/4225781392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:07.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:07.550 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:06.779922+0000 mgr.smithi080.vyzlfg (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 -- 172.21.15.112:0/4067345459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 msgr2=0x7f10500f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 --2- 172.21.15.112:0/4067345459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f10500f8030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1038009fb0 tx=0x7f1038009b30).stop 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 -- 172.21.15.112:0/4067345459 shutdown_connections 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 --2- 172.21.15.112:0/4067345459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f10500f8030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 -- 172.21.15.112:0/4067345459 >> 172.21.15.112:0/4067345459 conn(0x7f10500efe40 msgr2=0x7f10500f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 -- 172.21.15.112:0/4067345459 shutdown_connections 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.514+0000 7f1055659700 1 -- 172.21.15.112:0/4067345459 wait complete. 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f1055659700 1 Processor -- start 2024-02-13T03:33:09.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f1055659700 1 -- start start 2024-02-13T03:33:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f1055659700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f1055659700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050106d00 con 0x7f10500f5c40 2024-02-13T03:33:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f104effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f104effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47196/0 (socket says 172.21.15.112:47196) 2024-02-13T03:33:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.515+0000 7f104effd700 1 -- 172.21.15.112:0/2107449687 learned_addr learned my addr 172.21.15.112:0/2107449687 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:09.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f104effd700 1 -- 172.21.15.112:0/2107449687 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1038005040 con 0x7f10500f5c40 2024-02-13T03:33:09.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f104effd700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1038008a20 tx=0x7f103800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:09.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1038009320 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1038013380 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1050106f00 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103801eda0 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.516+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1050107390 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f103801e420 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1050045ad0 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f10477fe700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 0x7f103c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1038040ea0 con 0x7f10500f5c40 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f104e7fc700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 0x7f103c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:09.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.517+0000 7f104e7fc700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 0x7f103c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f10400096a0 tx=0x7f1040006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:09.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.521+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10380095f0 con 0x7f10500f5c40 2024-02-13T03:33:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.726+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1050108240 con 0x7f10500f5c40 2024-02-13T03:33:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.728+0000 7f10477fe700 1 -- 172.21.15.112:0/2107449687 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f103802c2d0 con 0x7f10500f5c40 2024-02-13T03:33:09.728 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:09.728 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 msgr2=0x7f103c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 0x7f103c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f10400096a0 tx=0x7f1040006b40).stop 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 msgr2=0x7f1050106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f1038008a20 tx=0x7f103800b040).stop 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 shutdown_connections 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f103c02ff60 0x7f103c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 --2- 172.21.15.112:0/2107449687 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f10500f5c40 0x7f1050106810 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 >> 172.21.15.112:0/2107449687 conn(0x7f10500efe40 msgr2=0x7f10500f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 shutdown_connections 2024-02-13T03:33:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:09.729+0000 7f1055659700 1 -- 172.21.15.112:0/2107449687 wait complete. 2024-02-13T03:33:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:08.780251+0000 mgr.smithi080.vyzlfg (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:10.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:09.728427+0000 mon.smithi080 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.112:0/2107449687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:11.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:11.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:10.780552+0000 mgr.smithi080.vyzlfg (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.329+0000 7efc1873d700 1 -- 172.21.15.112:0/1543294685 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 msgr2=0x7efc100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.329+0000 7efc1873d700 1 --2- 172.21.15.112:0/1543294685 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc100f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7efc00009fb0 tx=0x7efc00009b30).stop 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- 172.21.15.112:0/1543294685 shutdown_connections 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 --2- 172.21.15.112:0/1543294685 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc100f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- 172.21.15.112:0/1543294685 >> 172.21.15.112:0/1543294685 conn(0x7efc100f0b70 msgr2=0x7efc100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- 172.21.15.112:0/1543294685 shutdown_connections 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- 172.21.15.112:0/1543294685 wait complete. 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 Processor -- start 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- start start 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.330+0000 7efc1873d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10105bd0 con 0x7efc100f5410 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc164d9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc164d9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37308/0 (socket says 172.21.15.112:37308) 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc164d9700 1 -- 172.21.15.112:0/3032397368 learned_addr learned my addr 172.21.15.112:0/3032397368 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc164d9700 1 -- 172.21.15.112:0/3032397368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00005040 con 0x7efc100f5410 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc164d9700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efc00005010 tx=0x7efc0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc00009320 con 0x7efc100f5410 2024-02-13T03:33:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc10105dd0 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc00013380 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc00014e50 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.331+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc10106260 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.332+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc10045ad0 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.332+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc000095f0 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.332+0000 7efc06ffd700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.332+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc00040fa0 con 0x7efc100f5410 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.332+0000 7efc15cd8700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.333+0000 7efc15cd8700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 0x7efbfc0323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efc0c0096a0 tx=0x7efc0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.337+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc00014420 con 0x7efc100f5410 2024-02-13T03:33:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.542+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc10107110 con 0x7efc100f5410 2024-02-13T03:33:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.543+0000 7efc06ffd700 1 -- 172.21.15.112:0/3032397368 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc00010230 con 0x7efc100f5410 2024-02-13T03:33:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:13.543 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 msgr2=0x7efbfc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 0x7efbfc0323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efc0c0096a0 tx=0x7efc0c006b40).stop 2024-02-13T03:33:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 msgr2=0x7efc101056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:13.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efc00005010 tx=0x7efc0000b040).stop 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 shutdown_connections 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 --2- 172.21.15.112:0/3032397368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc100f5410 0x7efc101056e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 >> 172.21.15.112:0/3032397368 conn(0x7efc100f0b70 msgr2=0x7efc100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 shutdown_connections 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:13.544+0000 7efc1873d700 1 -- 172.21.15.112:0/3032397368 wait complete. 2024-02-13T03:33:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:12.780845+0000 mgr.smithi080.vyzlfg (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:14.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:13.543241+0000 mon.smithi080 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.112:0/3032397368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:15.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:15.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:14.781194+0000 mgr.smithi080.vyzlfg (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:17.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 -- 172.21.15.112:0/1720157008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 msgr2=0x7f70700f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 --2- 172.21.15.112:0/1720157008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f70700f90f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f7060009fb0 tx=0x7f7060009b30).stop 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 -- 172.21.15.112:0/1720157008 shutdown_connections 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 --2- 172.21.15.112:0/1720157008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f70700f90f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 -- 172.21.15.112:0/1720157008 >> 172.21.15.112:0/1720157008 conn(0x7f70700f0b70 msgr2=0x7f70700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 -- 172.21.15.112:0/1720157008 shutdown_connections 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.054+0000 7f7075cfd700 1 -- 172.21.15.112:0/1720157008 wait complete. 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f7075cfd700 1 Processor -- start 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f7075cfd700 1 -- start start 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f7075cfd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f7075cfd700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070106c70 con 0x7f7070060280 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f706f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f706f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37322/0 (socket says 172.21.15.112:37322) 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.055+0000 7f706f7fe700 1 -- 172.21.15.112:0/2121179216 learned_addr learned my addr 172.21.15.112:0/2121179216 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f706f7fe700 1 -- 172.21.15.112:0/2121179216 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060005040 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f706f7fe700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f70600089f0 tx=0x7f706000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7060009320 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7070106e70 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7060013380 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7060014da0 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.056+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070107300 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70600095f0 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7070045ad0 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f7067fff700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 0x7f705c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7060040de0 con 0x7f7070060280 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f706effd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 0x7f705c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.057+0000 7f706effd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 0x7f705c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f70580096a0 tx=0x7f7058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.062+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7060014420 con 0x7f7070060280 2024-02-13T03:33:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.271+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70701081b0 con 0x7f7070060280 2024-02-13T03:33:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.271+0000 7f7067fff700 1 -- 172.21.15.112:0/2121179216 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f706001c920 con 0x7f7070060280 2024-02-13T03:33:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:17.271 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 msgr2=0x7f705c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 0x7f705c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f70580096a0 tx=0x7f7058006b40).stop 2024-02-13T03:33:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 msgr2=0x7f7070106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f70600089f0 tx=0x7f706000b040).stop 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 shutdown_connections 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f705c02ff60 0x7f705c032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 --2- 172.21.15.112:0/2121179216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7070060280 0x7f7070106780 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 >> 172.21.15.112:0/2121179216 conn(0x7f70700f0b70 msgr2=0x7f70700f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 shutdown_connections 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:17.273+0000 7f7075cfd700 1 -- 172.21.15.112:0/2121179216 wait complete. 2024-02-13T03:33:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:17.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:17.271618+0000 mon.smithi080 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.112:0/2121179216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:16.781538+0000 mgr.smithi080.vyzlfg (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:19.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:19.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:20.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:18.781903+0000 mgr.smithi080.vyzlfg (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:21.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 -- 172.21.15.112:0/1335987182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 msgr2=0x7f1d180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:21.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 --2- 172.21.15.112:0/1335987182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d180f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f1d14009fb0 tx=0x7f1d14009b30).stop 2024-02-13T03:33:21.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 -- 172.21.15.112:0/1335987182 shutdown_connections 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 --2- 172.21.15.112:0/1335987182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d180f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 -- 172.21.15.112:0/1335987182 >> 172.21.15.112:0/1335987182 conn(0x7f1d180f1c70 msgr2=0x7f1d180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 -- 172.21.15.112:0/1335987182 shutdown_connections 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.056+0000 7f1d20438700 1 -- 172.21.15.112:0/1335987182 wait complete. 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d20438700 1 Processor -- start 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d20438700 1 -- start start 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d20438700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d20438700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d18106cd0 con 0x7f1d180f6510 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d1e1d4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d1e1d4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37344/0 (socket says 172.21.15.112:37344) 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d1e1d4700 1 -- 172.21.15.112:0/3371361570 learned_addr learned my addr 172.21.15.112:0/3371361570 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d1e1d4700 1 -- 172.21.15.112:0/3371361570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d14005040 con 0x7f1d180f6510 2024-02-13T03:33:21.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.057+0000 7f1d1e1d4700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1d14005010 tx=0x7f1d1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:21.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d14009320 con 0x7f1d180f6510 2024-02-13T03:33:21.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d18106ed0 con 0x7f1d180f6510 2024-02-13T03:33:21.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d14013380 con 0x7f1d180f6510 2024-02-13T03:33:21.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d14014e50 con 0x7f1d180f6510 2024-02-13T03:33:21.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d18107360 con 0x7f1d180f6510 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.058+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d140095f0 con 0x7f1d180f6510 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.059+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18045ad0 con 0x7f1d180f6510 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.059+0000 7f1d0effd700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.059+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d14040fa0 con 0x7f1d180f6510 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.059+0000 7f1d1d9d3700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:21.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.059+0000 7f1d1d9d3700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 0x7f1d040323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1d080096a0 tx=0x7f1d08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:21.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.064+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d1401d990 con 0x7f1d180f6510 2024-02-13T03:33:21.272 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.271+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d18108210 con 0x7f1d180f6510 2024-02-13T03:33:21.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.272+0000 7f1d0effd700 1 -- 172.21.15.112:0/3371361570 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d14010230 con 0x7f1d180f6510 2024-02-13T03:33:21.273 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:21.273 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:21.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 msgr2=0x7f1d040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 0x7f1d040323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1d080096a0 tx=0x7f1d08006b40).stop 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 msgr2=0x7f1d181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1d14005010 tx=0x7f1d1400b040).stop 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 shutdown_connections 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 --2- 172.21.15.112:0/3371361570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d180f6510 0x7f1d181067e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 >> 172.21.15.112:0/3371361570 conn(0x7f1d180f1c70 msgr2=0x7f1d180f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 shutdown_connections 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:21.274+0000 7f1d20438700 1 -- 172.21.15.112:0/3371361570 wait complete. 2024-02-13T03:33:21.275 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:21.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:21 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:21.273557+0000 mon.smithi080 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.112:0/3371361570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:22.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:22.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:20.782245+0000 mgr.smithi080.vyzlfg (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:22.782593+0000 mgr.smithi080.vyzlfg (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.952+0000 7fb678eb3700 1 -- 172.21.15.112:0/2545716753 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 msgr2=0x7fb6740f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.952+0000 7fb678eb3700 1 --2- 172.21.15.112:0/2545716753 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6740f4be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fb664009fb0 tx=0x7fb664009b30).stop 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- 172.21.15.112:0/2545716753 shutdown_connections 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 --2- 172.21.15.112:0/2545716753 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6740f4be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- 172.21.15.112:0/2545716753 >> 172.21.15.112:0/2545716753 conn(0x7fb6740f02e0 msgr2=0x7fb6740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- 172.21.15.112:0/2545716753 shutdown_connections 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- 172.21.15.112:0/2545716753 wait complete. 2024-02-13T03:33:24.955 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 Processor -- start 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- start start 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb678eb3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb674106cc0 con 0x7fb6740f47c0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb67259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb67259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54724/0 (socket says 172.21.15.112:54724) 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.953+0000 7fb67259c700 1 -- 172.21.15.112:0/3938972247 learned_addr learned my addr 172.21.15.112:0/3938972247 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb67259c700 1 -- 172.21.15.112:0/3938972247 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb664005040 con 0x7fb6740f47c0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb67259c700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb6640089f0 tx=0x7fb66400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb664009320 con 0x7fb6740f47c0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb674106ec0 con 0x7fb6740f47c0 2024-02-13T03:33:24.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb664013380 con 0x7fb6740f47c0 2024-02-13T03:33:24.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66401ed80 con 0x7fb6740f47c0 2024-02-13T03:33:24.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.954+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb674107350 con 0x7fb6740f47c0 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.955+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb66401e420 con 0x7fb6740f47c0 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.955+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb674045ad0 con 0x7fb6740f47c0 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.955+0000 7fb66affd700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.955+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb664040ed0 con 0x7fb6740f47c0 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.955+0000 7fb671d9b700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:24.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.956+0000 7fb671d9b700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 0x7fb660032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb65c0096a0 tx=0x7fb65c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:24.960 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:24.959+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb66401c930 con 0x7fb6740f47c0 2024-02-13T03:33:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.166+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb674108200 con 0x7fb6740f47c0 2024-02-13T03:33:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.167+0000 7fb66affd700 1 -- 172.21.15.112:0/3938972247 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb66401cb40 con 0x7fb6740f47c0 2024-02-13T03:33:25.167 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:25.168 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 msgr2=0x7fb660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 0x7fb660032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb65c0096a0 tx=0x7fb65c006b40).stop 2024-02-13T03:33:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 msgr2=0x7fb6741067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb6640089f0 tx=0x7fb66400b040).stop 2024-02-13T03:33:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 shutdown_connections 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 --2- 172.21.15.112:0/3938972247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6740f47c0 0x7fb6741067d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 >> 172.21.15.112:0/3938972247 conn(0x7fb6740f02e0 msgr2=0x7fb6740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.168+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 shutdown_connections 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:25.169+0000 7fb678eb3700 1 -- 172.21.15.112:0/3938972247 wait complete. 2024-02-13T03:33:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:25.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:25 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:25.167882+0000 mon.smithi080 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.112:0/3938972247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:24.782969+0000 mgr.smithi080.vyzlfg (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:26.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:26.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:26.783316+0000 mgr.smithi080.vyzlfg (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- 172.21.15.112:0/2734169058 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 msgr2=0x7f9a540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 --2- 172.21.15.112:0/2734169058 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a540f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9a48009fb0 tx=0x7f9a48009b30).stop 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- 172.21.15.112:0/2734169058 shutdown_connections 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 --2- 172.21.15.112:0/2734169058 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a540f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- 172.21.15.112:0/2734169058 >> 172.21.15.112:0/2734169058 conn(0x7f9a540f09a0 msgr2=0x7f9a540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- 172.21.15.112:0/2734169058 shutdown_connections 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- 172.21.15.112:0/2734169058 wait complete. 2024-02-13T03:33:28.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 Processor -- start 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- start start 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.975+0000 7f9a5359e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a541059c0 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54744/0 (socket says 172.21.15.112:54744) 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5259c700 1 -- 172.21.15.112:0/1442466186 learned_addr learned my addr 172.21.15.112:0/1442466186 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5259c700 1 -- 172.21.15.112:0/1442466186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48005040 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5259c700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9a480089f0 tx=0x7f9a4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a48009320 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a54105bc0 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a48013380 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a48014e20 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.976+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a54106050 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.977+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a54045ad0 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.977+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a480095f0 con 0x7f9a540f51a0 2024-02-13T03:33:28.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.977+0000 7f9a42ffd700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 0x7f9a3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:28.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.977+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a48042c70 con 0x7f9a540f51a0 2024-02-13T03:33:28.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.978+0000 7f9a51d9b700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 0x7f9a3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:28.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.978+0000 7f9a51d9b700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 0x7f9a3c0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9a440096a0 tx=0x7f9a44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:28.991 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:28.982+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a48014420 con 0x7f9a540f51a0 2024-02-13T03:33:29.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.190+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a54106f00 con 0x7f9a540f51a0 2024-02-13T03:33:29.191 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.191+0000 7f9a42ffd700 1 -- 172.21.15.112:0/1442466186 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a48011290 con 0x7f9a540f51a0 2024-02-13T03:33:29.192 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:29.192 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 msgr2=0x7f9a3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 0x7f9a3c0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9a440096a0 tx=0x7f9a44006b40).stop 2024-02-13T03:33:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 msgr2=0x7f9a541054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9a480089f0 tx=0x7f9a4800b040).stop 2024-02-13T03:33:29.193 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 shutdown_connections 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9a3c02ff10 0x7f9a3c0323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 --2- 172.21.15.112:0/1442466186 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9a540f51a0 0x7f9a541054d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 >> 172.21.15.112:0/1442466186 conn(0x7f9a540f09a0 msgr2=0x7f9a540f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 shutdown_connections 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:29.193+0000 7f9a5359e700 1 -- 172.21.15.112:0/1442466186 wait complete. 2024-02-13T03:33:29.194 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:29.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:29 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:29.192342+0000 mon.smithi080 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.112:0/1442466186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:30.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:30.818 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:28.783695+0000 mgr.smithi080.vyzlfg (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.798+0000 7fced8b44700 1 -- 172.21.15.112:0/2529899402 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 msgr2=0x7fced40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.798+0000 7fced8b44700 1 --2- 172.21.15.112:0/2529899402 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fcec8009fb0 tx=0x7fcec8009b30).stop 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- 172.21.15.112:0/2529899402 shutdown_connections 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 --2- 172.21.15.112:0/2529899402 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- 172.21.15.112:0/2529899402 >> 172.21.15.112:0/2529899402 conn(0x7fced40f0b70 msgr2=0x7fced40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- 172.21.15.112:0/2529899402 shutdown_connections 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- 172.21.15.112:0/2529899402 wait complete. 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 Processor -- start 2024-02-13T03:33:32.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- start start 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced8b44700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced40fb960 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:36270/0 (socket says 172.21.15.112:36270) 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.799+0000 7fced259c700 1 -- 172.21.15.112:0/554096410 learned_addr learned my addr 172.21.15.112:0/554096410 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fced259c700 1 -- 172.21.15.112:0/554096410 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec8005040 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fced259c700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fcec80089f0 tx=0x7fcec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec8009320 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced40fbb00 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcec8013380 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec801ed80 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.800+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced40fbf90 con 0x7fced40f5410 2024-02-13T03:33:32.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.801+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcec801e420 con 0x7fced40f5410 2024-02-13T03:33:32.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.801+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4059100 con 0x7fced40f5410 2024-02-13T03:33:32.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.801+0000 7fcec2ffd700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:32.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.801+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcec8041090 con 0x7fced40f5410 2024-02-13T03:33:32.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.801+0000 7fced1d9b700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:32.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.802+0000 7fced1d9b700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 0x7fcebc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fcec40096a0 tx=0x7fcec4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:32.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:32.806+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcec80095f0 con 0x7fced40f5410 2024-02-13T03:33:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:30.784006+0000 mgr.smithi080.vyzlfg (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:33.010 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.010+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fced40fafa0 con 0x7fced40f5410 2024-02-13T03:33:33.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.011+0000 7fcec2ffd700 1 -- 172.21.15.112:0/554096410 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcec8004490 con 0x7fced40f5410 2024-02-13T03:33:33.011 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:33.011 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 msgr2=0x7fcebc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 0x7fcebc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fcec40096a0 tx=0x7fcec4006b40).stop 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 msgr2=0x7fced40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fcec80089f0 tx=0x7fcec800b040).stop 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 shutdown_connections 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:33.013 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 --2- 172.21.15.112:0/554096410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fced40f5410 0x7fced40fd4e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:33.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 >> 172.21.15.112:0/554096410 conn(0x7fced40f0b70 msgr2=0x7fced405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:33.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 shutdown_connections 2024-02-13T03:33:33.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:33.012+0000 7fced8b44700 1 -- 172.21.15.112:0/554096410 wait complete. 2024-02-13T03:33:33.014 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:33.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:33.011775+0000 mon.smithi080 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.112:0/554096410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:34.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:34.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:32.784322+0000 mgr.smithi080.vyzlfg (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:36.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- 172.21.15.112:0/2567498967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 msgr2=0x7f62bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/2567498967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc0f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f62b0009fb0 tx=0x7f62b0009b30).stop 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- 172.21.15.112:0/2567498967 shutdown_connections 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/2567498967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc0f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- 172.21.15.112:0/2567498967 >> 172.21.15.112:0/2567498967 conn(0x7f62bc0f1c70 msgr2=0x7f62bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- 172.21.15.112:0/2567498967 shutdown_connections 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- 172.21.15.112:0/2567498967 wait complete. 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 Processor -- start 2024-02-13T03:33:36.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- start start 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.696+0000 7f62c2b6a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc106cd0 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c0906700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c0906700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:36286/0 (socket says 172.21.15.112:36286) 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c0906700 1 -- 172.21.15.112:0/1377940951 learned_addr learned my addr 172.21.15.112:0/1377940951 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c0906700 1 -- 172.21.15.112:0/1377940951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0005040 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c0906700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f62b0005010 tx=0x7f62b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b0009320 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc106ed0 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62b0013380 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62b0014e50 con 0x7f62bc0f6510 2024-02-13T03:33:36.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.697+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc107360 con 0x7f62bc0f6510 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.698+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62b0014420 con 0x7f62bc0f6510 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.698+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc045ad0 con 0x7f62bc0f6510 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.698+0000 7f62b97fa700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 0x7f62ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.698+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62b0040f90 con 0x7f62bc0f6510 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.698+0000 7f62bbfff700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 0x7f62ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:36.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.699+0000 7f62bbfff700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 0x7f62ac0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f62a80096a0 tx=0x7f62a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:36.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.703+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62b0014790 con 0x7f62bc0f6510 2024-02-13T03:33:36.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.907+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62bc108210 con 0x7f62bc0f6510 2024-02-13T03:33:36.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.907+0000 7f62b97fa700 1 -- 172.21.15.112:0/1377940951 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62b000e2a0 con 0x7f62bc0f6510 2024-02-13T03:33:36.908 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:36.908 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:36.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 msgr2=0x7f62ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:36.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 0x7f62ac0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f62a80096a0 tx=0x7f62a8006b40).stop 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 msgr2=0x7f62bc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f62b0005010 tx=0x7f62b000b040).stop 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 shutdown_connections 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f62ac02ff10 0x7f62ac0323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 --2- 172.21.15.112:0/1377940951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62bc0f6510 0x7f62bc1067e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 >> 172.21.15.112:0/1377940951 conn(0x7f62bc0f1c70 msgr2=0x7f62bc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 shutdown_connections 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:36.909+0000 7f62c2b6a700 1 -- 172.21.15.112:0/1377940951 wait complete. 2024-02-13T03:33:36.910 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:34.784655+0000 mgr.smithi080.vyzlfg (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:37.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:37 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:36.908754+0000 mon.smithi080 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.112:0/1377940951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:38.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:38.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:36.785009+0000 mgr.smithi080.vyzlfg (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- 172.21.15.112:0/3356741868 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 msgr2=0x7ff08c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 --2- 172.21.15.112:0/3356741868 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c0f66c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7ff088009fb0 tx=0x7ff088009b30).stop 2024-02-13T03:33:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- 172.21.15.112:0/3356741868 shutdown_connections 2024-02-13T03:33:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 --2- 172.21.15.112:0/3356741868 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c0f66c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- 172.21.15.112:0/3356741868 >> 172.21.15.112:0/3356741868 conn(0x7ff08c0f1aa0 msgr2=0x7ff08c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- 172.21.15.112:0/3356741868 shutdown_connections 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- 172.21.15.112:0/3356741868 wait complete. 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 Processor -- start 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.417+0000 7ff0922d0700 1 -- start start 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0922d0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0922d0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c106ac0 con 0x7ff08c0f62a0 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0912ce700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0912ce700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:36306/0 (socket says 172.21.15.112:36306) 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0912ce700 1 -- 172.21.15.112:0/2543666587 learned_addr learned my addr 172.21.15.112:0/2543666587 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0912ce700 1 -- 172.21.15.112:0/2543666587 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088005040 con 0x7ff08c0f62a0 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0912ce700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff0880089f0 tx=0x7ff08800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff088009320 con 0x7ff08c0f62a0 2024-02-13T03:33:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.418+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c106cc0 con 0x7ff08c0f62a0 2024-02-13T03:33:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c107150 con 0x7ff08c0f62a0 2024-02-13T03:33:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff088013380 con 0x7ff08c0f62a0 2024-02-13T03:33:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff088014e40 con 0x7ff08c0f62a0 2024-02-13T03:33:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff088014420 con 0x7ff08c0f62a0 2024-02-13T03:33:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c045ad0 con 0x7ff08c0f62a0 2024-02-13T03:33:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.419+0000 7ff081ffb700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 0x7ff0780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.420+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff088040f50 con 0x7ff08c0f62a0 2024-02-13T03:33:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.420+0000 7ff090acd700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 0x7ff0780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.420+0000 7ff090acd700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 0x7ff0780323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff07c0096a0 tx=0x7ff07c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:40.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.424+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff08806d2b0 con 0x7ff08c0f62a0 2024-02-13T03:33:40.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.629+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff08c108000 con 0x7ff08c0f62a0 2024-02-13T03:33:40.630 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.630+0000 7ff081ffb700 1 -- 172.21.15.112:0/2543666587 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff088012bb0 con 0x7ff08c0f62a0 2024-02-13T03:33:40.631 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:40.631 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 msgr2=0x7ff0780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 0x7ff0780323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff07c0096a0 tx=0x7ff07c006b40).stop 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 msgr2=0x7ff08c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff0880089f0 tx=0x7ff08800b040).stop 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 shutdown_connections 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff07802ff10 0x7ff0780323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 --2- 172.21.15.112:0/2543666587 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff08c0f62a0 0x7ff08c1065d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 >> 172.21.15.112:0/2543666587 conn(0x7ff08c0f1aa0 msgr2=0x7ff08c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 shutdown_connections 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:40.632+0000 7ff0922d0700 1 -- 172.21.15.112:0/2543666587 wait complete. 2024-02-13T03:33:40.633 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:38.785349+0000 mgr.smithi080.vyzlfg (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:41.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:40.631402+0000 mon.smithi080 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.112:0/2543666587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:42.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:42.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:40.785630+0000 mgr.smithi080.vyzlfg (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:44.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 -- 172.21.15.112:0/3665759980 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:44.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/3665759980 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1df8009fb0 tx=0x7f1df8009b30).stop 2024-02-13T03:33:44.194 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 -- 172.21.15.112:0/3665759980 shutdown_connections 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/3665759980 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e080f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 -- 172.21.15.112:0/3665759980 >> 172.21.15.112:0/3665759980 conn(0x7f1e080f0b70 msgr2=0x7f1e080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 -- 172.21.15.112:0/3665759980 shutdown_connections 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.169+0000 7f1e0f2df700 1 -- 172.21.15.112:0/3665759980 wait complete. 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0f2df700 1 Processor -- start 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0f2df700 1 -- start start 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0f2df700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0f2df700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08104ac0 con 0x7f1e080f5410 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0d07b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0d07b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:49784/0 (socket says 172.21.15.112:49784) 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.170+0000 7f1e0d07b700 1 -- 172.21.15.112:0/1621644946 learned_addr learned my addr 172.21.15.112:0/1621644946 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1e0d07b700 1 -- 172.21.15.112:0/1621644946 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8005040 con 0x7f1e080f5410 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1e0d07b700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1df8005010 tx=0x7f1df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8009320 con 0x7f1e080f5410 2024-02-13T03:33:44.195 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e08104cc0 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1df8013380 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df8014e50 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.171+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e08105150 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.172+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08045ad0 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.172+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1df80095f0 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.172+0000 7f1dfdffb700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 0x7f1df4034610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.172+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1df8040e60 con 0x7f1e080f5410 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.172+0000 7f1e0c87a700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 0x7f1df4034610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.173+0000 7f1e0c87a700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 0x7f1df4034610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1e040096a0 tx=0x7f1e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:44.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.177+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1df8014420 con 0x7f1e080f5410 2024-02-13T03:33:44.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.383+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e08105fa0 con 0x7f1e080f5410 2024-02-13T03:33:44.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.384+0000 7f1dfdffb700 1 -- 172.21.15.112:0/1621644946 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1df8010230 con 0x7f1e080f5410 2024-02-13T03:33:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:44.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.385+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 msgr2=0x7f1df4034610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.385+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 0x7f1df4034610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1e040096a0 tx=0x7f1e04006b40).stop 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.385+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 msgr2=0x7f1e081045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.385+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1df8005010 tx=0x7f1df800b040).stop 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 shutdown_connections 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1df4032150 0x7f1df4034610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 --2- 172.21.15.112:0/1621644946 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e080f5410 0x7f1e081045d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:44.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 >> 172.21.15.112:0/1621644946 conn(0x7f1e080f0b70 msgr2=0x7f1e080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:44.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 shutdown_connections 2024-02-13T03:33:44.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:44.386+0000 7f1e0f2df700 1 -- 172.21.15.112:0/1621644946 wait complete. 2024-02-13T03:33:44.387 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:42.785980+0000 mgr.smithi080.vyzlfg (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:44.384834+0000 mon.smithi080 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.112:0/1621644946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:46.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:46.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:44.786354+0000 mgr.smithi080.vyzlfg (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 -- 172.21.15.112:0/3434156302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 msgr2=0x7f8c1c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 --2- 172.21.15.112:0/3434156302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c0f66c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8c10009fb0 tx=0x7f8c10009b30).stop 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 -- 172.21.15.112:0/3434156302 shutdown_connections 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 --2- 172.21.15.112:0/3434156302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c0f66c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 -- 172.21.15.112:0/3434156302 >> 172.21.15.112:0/3434156302 conn(0x7f8c1c0f1ac0 msgr2=0x7f8c1c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:47.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 -- 172.21.15.112:0/3434156302 shutdown_connections 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.928+0000 7f8c219b8700 1 -- 172.21.15.112:0/3434156302 wait complete. 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c219b8700 1 Processor -- start 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c219b8700 1 -- start start 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c219b8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c219b8700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c069e20 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c209b6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c209b6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:49804/0 (socket says 172.21.15.112:49804) 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c209b6700 1 -- 172.21.15.112:0/1565849059 learned_addr learned my addr 172.21.15.112:0/1565849059 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.929+0000 7f8c209b6700 1 -- 172.21.15.112:0/1565849059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10005040 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c209b6700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8c100089f0 tx=0x7f8c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c10009320 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c069fc0 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c10013380 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c10014e20 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.957 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c06a450 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.930+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c100095f0 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.931+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1c059100 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.931+0000 7f8c197fa700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 0x7f8c0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.931+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c10040f50 con 0x7f8c1c0f62a0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.931+0000 7f8c1bfff700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 0x7f8c0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.931+0000 7f8c1bfff700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 0x7f8c0c0323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8c080096a0 tx=0x7f8c08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:47.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:47.936+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c10014420 con 0x7f8c1c0f62a0 2024-02-13T03:33:48.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.139+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c1c0fbe30 con 0x7f8c1c0f62a0 2024-02-13T03:33:48.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c197fa700 1 -- 172.21.15.112:0/1565849059 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c1001d980 con 0x7f8c1c0f62a0 2024-02-13T03:33:48.141 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:48.141 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 msgr2=0x7f8c0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 0x7f8c0c0323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8c080096a0 tx=0x7f8c08006b40).stop 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 msgr2=0x7f8c1c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f8c100089f0 tx=0x7f8c1000b040).stop 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 shutdown_connections 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8c0c02ff10 0x7f8c0c0323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 --2- 172.21.15.112:0/1565849059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8c1c0f62a0 0x7f8c1c06b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 >> 172.21.15.112:0/1565849059 conn(0x7f8c1c0f1ac0 msgr2=0x7f8c1c0f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 shutdown_connections 2024-02-13T03:33:48.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:48.141+0000 7f8c219b8700 1 -- 172.21.15.112:0/1565849059 wait complete. 2024-02-13T03:33:48.143 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:46.786749+0000 mgr.smithi080.vyzlfg (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:48.141299+0000 mon.smithi080 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.112:0/1565849059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:49.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:49.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:48.787167+0000 mgr.smithi080.vyzlfg (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 -- 172.21.15.112:0/3281300827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 msgr2=0x7effc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 --2- 172.21.15.112:0/3281300827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc40f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7effac009fb0 tx=0x7effac009b30).stop 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 -- 172.21.15.112:0/3281300827 shutdown_connections 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 --2- 172.21.15.112:0/3281300827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc40f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 -- 172.21.15.112:0/3281300827 >> 172.21.15.112:0/3281300827 conn(0x7effc40f0b70 msgr2=0x7effc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 -- 172.21.15.112:0/3281300827 shutdown_connections 2024-02-13T03:33:51.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.752+0000 7effc9764700 1 -- 172.21.15.112:0/3281300827 wait complete. 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc9764700 1 Processor -- start 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc9764700 1 -- start start 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc9764700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc9764700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4105bd0 con 0x7effc40f5410 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc2ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc2ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43922/0 (socket says 172.21.15.112:43922) 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.753+0000 7effc2ffd700 1 -- 172.21.15.112:0/2038728828 learned_addr learned my addr 172.21.15.112:0/2038728828 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effc2ffd700 1 -- 172.21.15.112:0/2038728828 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac005040 con 0x7effc40f5410 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effc2ffd700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7effac005010 tx=0x7effac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac009320 con 0x7effc40f5410 2024-02-13T03:33:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc4105dd0 con 0x7effc40f5410 2024-02-13T03:33:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effac013380 con 0x7effc40f5410 2024-02-13T03:33:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effac014e50 con 0x7effc40f5410 2024-02-13T03:33:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.754+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc4106260 con 0x7effc40f5410 2024-02-13T03:33:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.755+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effac0095f0 con 0x7effc40f5410 2024-02-13T03:33:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.755+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc4045ad0 con 0x7effc40f5410 2024-02-13T03:33:51.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.755+0000 7effbb7fe700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 0x7effb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:51.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.755+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effac042d80 con 0x7effc40f5410 2024-02-13T03:33:51.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.755+0000 7effc27fc700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 0x7effb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:51.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.756+0000 7effc27fc700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 0x7effb00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7effb40096a0 tx=0x7effb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:51.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.760+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effac014420 con 0x7effc40f5410 2024-02-13T03:33:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.964+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effc4107110 con 0x7effc40f5410 2024-02-13T03:33:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.965+0000 7effbb7fe700 1 -- 172.21.15.112:0/2038728828 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effac011290 con 0x7effc40f5410 2024-02-13T03:33:51.964 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:51.964 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 msgr2=0x7effb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 0x7effb00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7effb40096a0 tx=0x7effb4006b40).stop 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 msgr2=0x7effc41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7effac005010 tx=0x7effac00b040).stop 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 shutdown_connections 2024-02-13T03:33:51.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7effb002ff10 0x7effb00323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:51.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 --2- 172.21.15.112:0/2038728828 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7effc40f5410 0x7effc41056e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:51.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 >> 172.21.15.112:0/2038728828 conn(0x7effc40f0b70 msgr2=0x7effc40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:51.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 shutdown_connections 2024-02-13T03:33:51.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:51.966+0000 7effc9764700 1 -- 172.21.15.112:0/2038728828 wait complete. 2024-02-13T03:33:51.967 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:50.787510+0000 mgr.smithi080.vyzlfg (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:52.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:51.964927+0000 mon.smithi080 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.112:0/2038728828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:53.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:53.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:52.787843+0000 mgr.smithi080.vyzlfg (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:55.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 -- 172.21.15.112:0/2419054707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 msgr2=0x7fa1cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/2419054707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc0f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fa1c0009fb0 tx=0x7fa1c0009b30).stop 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 -- 172.21.15.112:0/2419054707 shutdown_connections 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/2419054707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc0f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 -- 172.21.15.112:0/2419054707 >> 172.21.15.112:0/2419054707 conn(0x7fa1cc0f0b70 msgr2=0x7fa1cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 -- 172.21.15.112:0/2419054707 shutdown_connections 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.534+0000 7fa1d31f3700 1 -- 172.21.15.112:0/2419054707 wait complete. 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d31f3700 1 Processor -- start 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d31f3700 1 -- start start 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d31f3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d31f3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc060240 con 0x7fa1cc0f5410 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d0f8f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d0f8f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43930/0 (socket says 172.21.15.112:43930) 2024-02-13T03:33:55.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d0f8f700 1 -- 172.21.15.112:0/1576453747 learned_addr learned my addr 172.21.15.112:0/1576453747 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.535+0000 7fa1d0f8f700 1 -- 172.21.15.112:0/1576453747 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c0005040 con 0x7fa1cc0f5410 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1d0f8f700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fa1c0005010 tx=0x7fa1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1c0009320 con 0x7fa1cc0f5410 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc060440 con 0x7fa1cc0f5410 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1c0013380 con 0x7fa1cc0f5410 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1c0014e50 con 0x7fa1cc0f5410 2024-02-13T03:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc109b90 con 0x7fa1cc0f5410 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.536+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1c00095f0 con 0x7fa1cc0f5410 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.537+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc045ad0 con 0x7fa1cc0f5410 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.537+0000 7fa1c97fa700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 0x7fa1bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.537+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1c0040f10 con 0x7fa1cc0f5410 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.537+0000 7fa1cbfff700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 0x7fa1bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:55.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.537+0000 7fa1cbfff700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 0x7fa1bc0323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fa1b80096a0 tx=0x7fa1b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:55.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.541+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1c0014420 con 0x7fa1cc0f5410 2024-02-13T03:33:55.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.744+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1cc10a3b0 con 0x7fa1cc0f5410 2024-02-13T03:33:55.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.745+0000 7fa1c97fa700 1 -- 172.21.15.112:0/1576453747 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1c0010230 con 0x7fa1cc0f5410 2024-02-13T03:33:55.745 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:55.745 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:55.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 msgr2=0x7fa1bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:55.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 0x7fa1bc0323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fa1b80096a0 tx=0x7fa1b8006b40).stop 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 msgr2=0x7fa1cc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fa1c0005010 tx=0x7fa1c000b040).stop 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 shutdown_connections 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa1bc02ff10 0x7fa1bc0323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 --2- 172.21.15.112:0/1576453747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa1cc0f5410 0x7fa1cc1094e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 >> 172.21.15.112:0/1576453747 conn(0x7fa1cc0f0b70 msgr2=0x7fa1cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 shutdown_connections 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:55.747+0000 7fa1d31f3700 1 -- 172.21.15.112:0/1576453747 wait complete. 2024-02-13T03:33:55.747 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:33:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:54.788207+0000 mgr.smithi080.vyzlfg (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:55.745332+0000 mon.smithi080 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.112:0/1576453747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:33:57.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:33:57.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:33:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:56.860385+0000 mon.smithi080 (mon.0) 239 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:33:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:56.867813+0000 mon.smithi080 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:33:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:33:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:56.788581+0000 mgr.smithi080.vyzlfg (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:33:59.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 -- 172.21.15.112:0/1992086237 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 msgr2=0x7f73a00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:59.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 --2- 172.21.15.112:0/1992086237 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a00fb570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7390009fb0 tx=0x7f7390009b30).stop 2024-02-13T03:33:59.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 -- 172.21.15.112:0/1992086237 shutdown_connections 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 --2- 172.21.15.112:0/1992086237 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a00fb570 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 -- 172.21.15.112:0/1992086237 >> 172.21.15.112:0/1992086237 conn(0x7f73a00f0b70 msgr2=0x7f73a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 -- 172.21.15.112:0/1992086237 shutdown_connections 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.397+0000 7f73a7412700 1 -- 172.21.15.112:0/1992086237 wait complete. 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a7412700 1 Processor -- start 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a7412700 1 -- start start 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a7412700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a7412700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0108e90 con 0x7f73a0060200 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a51ae700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a51ae700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43958/0 (socket says 172.21.15.112:43958) 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.398+0000 7f73a51ae700 1 -- 172.21.15.112:0/891301040 learned_addr learned my addr 172.21.15.112:0/891301040 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f73a51ae700 1 -- 172.21.15.112:0/891301040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390005040 con 0x7f73a0060200 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f73a51ae700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7390009830 tx=0x7f739000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:59.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7390009320 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7390013380 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a0109090 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7390014420 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.399+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0109520 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.400+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a0045ad0 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.400+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f739001e420 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.400+0000 7f7395ffb700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 0x7f738c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.400+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7390040e30 con 0x7f73a0060200 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.400+0000 7f73a49ad700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 0x7f738c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.401+0000 7f73a49ad700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 0x7f738c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f739c0096a0 tx=0x7f739c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:33:59.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.405+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f739001cb10 con 0x7f73a0060200 2024-02-13T03:33:59.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.612+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73a00fe290 con 0x7f73a0060200 2024-02-13T03:33:59.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.612+0000 7f7395ffb700 1 -- 172.21.15.112:0/891301040 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7390004490 con 0x7f73a0060200 2024-02-13T03:33:59.613 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:33:59.613 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 msgr2=0x7f738c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 0x7f738c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f739c0096a0 tx=0x7f739c006b40).stop 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 msgr2=0x7f73a01089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7390009830 tx=0x7f739000b040).stop 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 shutdown_connections 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f738c02ff60 0x7f738c032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 --2- 172.21.15.112:0/891301040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f73a0060200 0x7f73a01089a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 >> 172.21.15.112:0/891301040 conn(0x7f73a00f0b70 msgr2=0x7f73a00f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 shutdown_connections 2024-02-13T03:33:59.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:33:59.614+0000 7f73a7412700 1 -- 172.21.15.112:0/891301040 wait complete. 2024-02-13T03:33:59.616 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:33:58.788834+0000 mgr.smithi080.vyzlfg (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:33:59.613730+0000 mon.smithi080 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.112:0/891301040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:01.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:01.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:00.789180+0000 mgr.smithi080.vyzlfg (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:03.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/802668059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 msgr2=0x7f5e880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/802668059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e880f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f5e78009fb0 tx=0x7f5e78009b30).stop 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/802668059 shutdown_connections 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/802668059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e880f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/802668059 >> 172.21.15.112:0/802668059 conn(0x7f5e880f1c70 msgr2=0x7f5e880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/802668059 shutdown_connections 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.262+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/802668059 wait complete. 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8fc6c700 1 Processor -- start 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8fc6c700 1 -- start start 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8fc6c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8fc6c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88069e20 con 0x7f5e880f6510 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8da08700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:03.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8da08700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42742/0 (socket says 172.21.15.112:42742) 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8da08700 1 -- 172.21.15.112:0/1723838839 learned_addr learned my addr 172.21.15.112:0/1723838839 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.263+0000 7f5e8da08700 1 -- 172.21.15.112:0/1723838839 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78005040 con 0x7f5e880f6510 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e8da08700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f5e78005010 tx=0x7f5e7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e78009320 con 0x7f5e880f6510 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e88069fc0 con 0x7f5e880f6510 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e78013380 con 0x7f5e880f6510 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e78014e50 con 0x7f5e880f6510 2024-02-13T03:34:03.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8806a450 con 0x7f5e880f6510 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.264+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e780095f0 con 0x7f5e880f6510 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.265+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88059100 con 0x7f5e880f6510 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.265+0000 7f5e7e7fc700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 0x7f5e740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.265+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e78040e90 con 0x7f5e880f6510 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.265+0000 7f5e8d207700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 0x7f5e740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:03.267 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.265+0000 7f5e8d207700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 0x7f5e740323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5e840096a0 tx=0x7f5e84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:03.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.270+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e78014420 con 0x7f5e880f6510 2024-02-13T03:34:03.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.474+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e88045ad0 con 0x7f5e880f6510 2024-02-13T03:34:03.476 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.475+0000 7f5e7e7fc700 1 -- 172.21.15.112:0/1723838839 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e78010230 con 0x7f5e880f6510 2024-02-13T03:34:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:03.476 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 msgr2=0x7f5e740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 0x7f5e740323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5e840096a0 tx=0x7f5e84006b40).stop 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 msgr2=0x7f5e8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f5e78005010 tx=0x7f5e7800b040).stop 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 shutdown_connections 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e7402ff10 0x7f5e740323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 --2- 172.21.15.112:0/1723838839 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e880f6510 0x7f5e8806b9a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 >> 172.21.15.112:0/1723838839 conn(0x7f5e880f1c70 msgr2=0x7f5e880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 shutdown_connections 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:03.477+0000 7f5e8fc6c700 1 -- 172.21.15.112:0/1723838839 wait complete. 2024-02-13T03:34:03.478 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:03.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:03 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:03.476527+0000 mon.smithi080 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.112:0/1723838839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:02.789509+0000 mgr.smithi080.vyzlfg (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:05.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:05.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:04.789806+0000 mgr.smithi080.vyzlfg (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:07.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 -- 172.21.15.112:0/211847157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 msgr2=0x7fa9200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 --2- 172.21.15.112:0/211847157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa9200f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa908009fb0 tx=0x7fa908009b30).stop 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 -- 172.21.15.112:0/211847157 shutdown_connections 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 --2- 172.21.15.112:0/211847157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa9200f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 -- 172.21.15.112:0/211847157 >> 172.21.15.112:0/211847157 conn(0x7fa9200f1c70 msgr2=0x7fa9200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 -- 172.21.15.112:0/211847157 shutdown_connections 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.171+0000 7fa9267fe700 1 -- 172.21.15.112:0/211847157 wait complete. 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa9267fe700 1 Processor -- start 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa9267fe700 1 -- start start 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa9267fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa9267fe700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920069e20 con 0x7fa9200f6510 2024-02-13T03:34:07.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:07.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa91ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42758/0 (socket says 172.21.15.112:42758) 2024-02-13T03:34:07.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa91ffff700 1 -- 172.21.15.112:0/1594747726 learned_addr learned my addr 172.21.15.112:0/1594747726 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:07.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.172+0000 7fa91ffff700 1 -- 172.21.15.112:0/1594747726 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908005040 con 0x7fa9200f6510 2024-02-13T03:34:07.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa91ffff700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fa908005010 tx=0x7fa90800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:07.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908009320 con 0x7fa9200f6510 2024-02-13T03:34:07.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa920069fc0 con 0x7fa9200f6510 2024-02-13T03:34:07.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa908013380 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908014e50 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.173+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92006a450 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.174+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9080263a0 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.174+0000 7fa91cff9700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 0x7fa90c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.174+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa908040f10 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.174+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa920059100 con 0x7fa9200f6510 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.174+0000 7fa91f7fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 0x7fa90c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:07.176 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.175+0000 7fa91f7fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 0x7fa90c0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa9100096a0 tx=0x7fa910006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:07.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.179+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa908012920 con 0x7fa9200f6510 2024-02-13T03:34:07.383 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.383+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa92006ab50 con 0x7fa9200f6510 2024-02-13T03:34:07.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.384+0000 7fa91cff9700 1 -- 172.21.15.112:0/1594747726 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa90802b020 con 0x7fa9200f6510 2024-02-13T03:34:07.384 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:07.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 msgr2=0x7fa90c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 0x7fa90c0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa9100096a0 tx=0x7fa910006b40).stop 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 msgr2=0x7fa92006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fa908005010 tx=0x7fa90800b040).stop 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 shutdown_connections 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa90c02ff10 0x7fa90c0323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:07.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 --2- 172.21.15.112:0/1594747726 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:07.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 >> 172.21.15.112:0/1594747726 conn(0x7fa9200f1c70 msgr2=0x7fa9200f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:07.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 shutdown_connections 2024-02-13T03:34:07.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:07.385+0000 7fa9267fe700 1 -- 172.21.15.112:0/1594747726 wait complete. 2024-02-13T03:34:07.387 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:07.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:07.384907+0000 mon.smithi080 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.112:0/1594747726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:06.790154+0000 mgr.smithi080.vyzlfg (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:09.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:09.023 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:08.790469+0000 mgr.smithi080.vyzlfg (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:11.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 -- 172.21.15.112:0/3408839800 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 msgr2=0x7fd20c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:11.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 --2- 172.21.15.112:0/3408839800 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fd200009fb0 tx=0x7fd200009b30).stop 2024-02-13T03:34:11.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 -- 172.21.15.112:0/3408839800 shutdown_connections 2024-02-13T03:34:11.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 --2- 172.21.15.112:0/3408839800 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:11.014 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 -- 172.21.15.112:0/3408839800 >> 172.21.15.112:0/3408839800 conn(0x7fd20c0f1c70 msgr2=0x7fd20c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 -- 172.21.15.112:0/3408839800 shutdown_connections 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.012+0000 7fd21234c700 1 -- 172.21.15.112:0/3408839800 wait complete. 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd21234c700 1 Processor -- start 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd21234c700 1 -- start start 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd21234c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd21234c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c069e20 con 0x7fd20c0f6510 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd20bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd20bfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42782/0 (socket says 172.21.15.112:42782) 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.013+0000 7fd20bfff700 1 -- 172.21.15.112:0/1839578943 learned_addr learned my addr 172.21.15.112:0/1839578943 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd20bfff700 1 -- 172.21.15.112:0/1839578943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200005040 con 0x7fd20c0f6510 2024-02-13T03:34:11.015 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd20bfff700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd200005010 tx=0x7fd20000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:11.016 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd200009320 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20c069fc0 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd200013380 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd200014e50 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.014+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20c06a450 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.015+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2000095f0 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.015+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20c059100 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.015+0000 7fd208ff9700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 0x7fd1f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.015+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd200040f70 con 0x7fd20c0f6510 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.015+0000 7fd20b7fe700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 0x7fd1f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:11.017 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.016+0000 7fd20b7fe700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 0x7fd1f40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd1fc0096a0 tx=0x7fd1fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:11.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.020+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd200014420 con 0x7fd20c0f6510 2024-02-13T03:34:11.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.228+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd20c045ad0 con 0x7fd20c0f6510 2024-02-13T03:34:11.229 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.229+0000 7fd208ff9700 1 -- 172.21.15.112:0/1839578943 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd200010230 con 0x7fd20c0f6510 2024-02-13T03:34:11.230 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:11.230 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:11.231 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.230+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 msgr2=0x7fd1f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.230+0000 7fd21234c700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 0x7fd1f40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd1fc0096a0 tx=0x7fd1fc006b40).stop 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.230+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 msgr2=0x7fd20c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.230+0000 7fd21234c700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd200005010 tx=0x7fd20000b040).stop 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 shutdown_connections 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd1f402ff10 0x7fd1f40323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 --2- 172.21.15.112:0/1839578943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd20c0f6510 0x7fd20c06b9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 >> 172.21.15.112:0/1839578943 conn(0x7fd20c0f1c70 msgr2=0x7fd20c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 shutdown_connections 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:11.231+0000 7fd21234c700 1 -- 172.21.15.112:0/1839578943 wait complete. 2024-02-13T03:34:11.232 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:11.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:11 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:11.230060+0000 mon.smithi080 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.112:0/1839578943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:10.790811+0000 mgr.smithi080.vyzlfg (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:13.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:13.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:12.791051+0000 mgr.smithi080.vyzlfg (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.035+0000 7f567759e700 1 -- 172.21.15.112:0/966790209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 msgr2=0x7f56780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.035+0000 7f567759e700 1 --2- 172.21.15.112:0/966790209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f56780f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f566c009fb0 tx=0x7f566c009b30).stop 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- 172.21.15.112:0/966790209 shutdown_connections 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 --2- 172.21.15.112:0/966790209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f56780f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- 172.21.15.112:0/966790209 >> 172.21.15.112:0/966790209 conn(0x7f56780f09a0 msgr2=0x7f56780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:15.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- 172.21.15.112:0/966790209 shutdown_connections 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- 172.21.15.112:0/966790209 wait complete. 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 Processor -- start 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- start start 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.036+0000 7f567759e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5678060280 con 0x7f56780f51a0 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567659c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:44870/0 (socket says 172.21.15.112:44870) 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567659c700 1 -- 172.21.15.112:0/3984092994 learned_addr learned my addr 172.21.15.112:0/3984092994 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:15.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567659c700 1 -- 172.21.15.112:0/3984092994 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c005040 con 0x7f56780f51a0 2024-02-13T03:34:15.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567659c700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f566c0089f0 tx=0x7f566c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566c009320 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f566c013380 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566c014e20 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5678060480 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.037+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56781099c0 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.038+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f566c0095f0 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.038+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5678045ad0 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.038+0000 7f5666ffd700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 0x7f5660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.038+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f566c040e40 con 0x7f56780f51a0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.038+0000 7f5675d9b700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 0x7f5660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:15.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.039+0000 7f5675d9b700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 0x7f5660032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f56680096a0 tx=0x7f5668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:15.043 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.043+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f566c014420 con 0x7f56780f51a0 2024-02-13T03:34:15.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.246+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56780fad30 con 0x7f56780f51a0 2024-02-13T03:34:15.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.247+0000 7f5666ffd700 1 -- 172.21.15.112:0/3984092994 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f566c010230 con 0x7f56780f51a0 2024-02-13T03:34:15.248 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:15.248 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:15.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 msgr2=0x7f5660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:15.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 0x7f5660032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f56680096a0 tx=0x7f5668006b40).stop 2024-02-13T03:34:15.249 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 msgr2=0x7f5678109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f566c0089f0 tx=0x7f566c00b040).stop 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 shutdown_connections 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f566002ff60 0x7f5660032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 --2- 172.21.15.112:0/3984092994 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f56780f51a0 0x7f5678109290 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 >> 172.21.15.112:0/3984092994 conn(0x7f56780f09a0 msgr2=0x7f56780f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 shutdown_connections 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:15.248+0000 7f567759e700 1 -- 172.21.15.112:0/3984092994 wait complete. 2024-02-13T03:34:15.250 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:15.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:15 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:15.248283+0000 mon.smithi080 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.112:0/3984092994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:14.791403+0000 mgr.smithi080.vyzlfg (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:17.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:17.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:16.791743+0000 mgr.smithi080.vyzlfg (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 -- 172.21.15.112:0/2081293617 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 msgr2=0x7f7ec80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 --2- 172.21.15.112:0/2081293617 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec80f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7ebc009fb0 tx=0x7f7ebc009b30).stop 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 -- 172.21.15.112:0/2081293617 shutdown_connections 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 --2- 172.21.15.112:0/2081293617 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec80f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 -- 172.21.15.112:0/2081293617 >> 172.21.15.112:0/2081293617 conn(0x7f7ec80f0b70 msgr2=0x7f7ec80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 -- 172.21.15.112:0/2081293617 shutdown_connections 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 -- 172.21.15.112:0/2081293617 wait complete. 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.948+0000 7f7ecede0700 1 Processor -- start 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7ecede0700 1 -- start start 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7ecede0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7ecede0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8106cd0 con 0x7f7ec80f5410 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7eccb7c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:18.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7eccb7c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:44886/0 (socket says 172.21.15.112:44886) 2024-02-13T03:34:18.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7eccb7c700 1 -- 172.21.15.112:0/1681050309 learned_addr learned my addr 172.21.15.112:0/1681050309 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:18.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7eccb7c700 1 -- 172.21.15.112:0/1681050309 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc005040 con 0x7f7ec80f5410 2024-02-13T03:34:18.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.949+0000 7f7eccb7c700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ebc005010 tx=0x7f7ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ebc009320 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ebc013380 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec8106ed0 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ebc014e50 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec8107360 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.950+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ebc0263a0 con 0x7f7ec80f5410 2024-02-13T03:34:18.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.951+0000 7f7ec57fa700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:18.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.951+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ebc040fb0 con 0x7f7ec80f5410 2024-02-13T03:34:18.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.951+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8045ad0 con 0x7f7ec80f5410 2024-02-13T03:34:18.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.951+0000 7f7ec7fff700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:18.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.951+0000 7f7ec7fff700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 0x7f7eb80323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7eb40096a0 tx=0x7f7eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:18.956 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:18.956+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ebc012920 con 0x7f7ec80f5410 2024-02-13T03:34:19.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.159+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ec8108080 con 0x7f7ec80f5410 2024-02-13T03:34:19.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.160+0000 7f7ec57fa700 1 -- 172.21.15.112:0/1681050309 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ebc012330 con 0x7f7ec80f5410 2024-02-13T03:34:19.161 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:19.161 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:19.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.161+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 msgr2=0x7f7eb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:19.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.161+0000 7f7ecede0700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 0x7f7eb80323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7eb40096a0 tx=0x7f7eb4006b40).stop 2024-02-13T03:34:19.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.161+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 msgr2=0x7f7ec81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:19.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.161+0000 7f7ecede0700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ebc005010 tx=0x7f7ebc00b040).stop 2024-02-13T03:34:19.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 shutdown_connections 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 --2- 172.21.15.112:0/1681050309 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ec80f5410 0x7f7ec81067e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 >> 172.21.15.112:0/1681050309 conn(0x7f7ec80f0b70 msgr2=0x7f7ec805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 shutdown_connections 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:19.162+0000 7f7ecede0700 1 -- 172.21.15.112:0/1681050309 wait complete. 2024-02-13T03:34:19.163 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:19.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:19 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:19.161310+0000 mon.smithi080 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.112:0/1681050309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:20.910 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:20.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:20.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:18.791978+0000 mgr.smithi080.vyzlfg (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- 172.21.15.112:0/3430254368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 msgr2=0x7f08a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 --2- 172.21.15.112:0/3430254368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a00f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f0890009fb0 tx=0x7f0890009b30).stop 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- 172.21.15.112:0/3430254368 shutdown_connections 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 --2- 172.21.15.112:0/3430254368 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a00f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- 172.21.15.112:0/3430254368 >> 172.21.15.112:0/3430254368 conn(0x7f08a00f09a0 msgr2=0x7f08a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:22.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- 172.21.15.112:0/3430254368 shutdown_connections 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- 172.21.15.112:0/3430254368 wait complete. 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 Processor -- start 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 -- start start 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.863+0000 7f08a650d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a650d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a01059c0 con 0x7f08a00f51a0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a550b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a550b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34176/0 (socket says 172.21.15.112:34176) 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a550b700 1 -- 172.21.15.112:0/1929981140 learned_addr learned my addr 172.21.15.112:0/1929981140 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a550b700 1 -- 172.21.15.112:0/1929981140 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0890005040 con 0x7f08a00f51a0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a550b700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f08900089f0 tx=0x7f089000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0890009320 con 0x7f08a00f51a0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a0105bc0 con 0x7f08a00f51a0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0890013380 con 0x7f08a00f51a0 2024-02-13T03:34:22.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.864+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0890014e20 con 0x7f08a00f51a0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.865+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a0106050 con 0x7f08a00f51a0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.865+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08900095f0 con 0x7f08a00f51a0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.865+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a0045ad0 con 0x7f08a00f51a0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.865+0000 7f0895ffb700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 0x7f088c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.865+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0890040f10 con 0x7f08a00f51a0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.866+0000 7f08a4d0a700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 0x7f088c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.866+0000 7f08a4d0a700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 0x7f088c0323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f089c0096a0 tx=0x7f089c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:22.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:22.870+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0890014420 con 0x7f08a00f51a0 2024-02-13T03:34:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:20.792261+0000 mgr.smithi080.vyzlfg (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.073+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08a0106f00 con 0x7f08a00f51a0 2024-02-13T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.074+0000 7f0895ffb700 1 -- 172.21.15.112:0/1929981140 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f089001d980 con 0x7f08a00f51a0 2024-02-13T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:23.075 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 msgr2=0x7f088c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 0x7f088c0323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f089c0096a0 tx=0x7f089c006b40).stop 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 msgr2=0x7f08a01054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f08900089f0 tx=0x7f089000b040).stop 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 shutdown_connections 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f088c02ff10 0x7f088c0323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 --2- 172.21.15.112:0/1929981140 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f08a00f51a0 0x7f08a01054d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:23.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 >> 172.21.15.112:0/1929981140 conn(0x7f08a00f09a0 msgr2=0x7f08a00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:23.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 shutdown_connections 2024-02-13T03:34:23.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:23.075+0000 7f08a650d700 1 -- 172.21.15.112:0/1929981140 wait complete. 2024-02-13T03:34:23.077 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:23.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:23 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:23.075178+0000 mon.smithi080 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.112:0/1929981140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:24.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:24.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:22.792593+0000 mgr.smithi080.vyzlfg (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 -- 172.21.15.112:0/2413075696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 msgr2=0x7f2b400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 --2- 172.21.15.112:0/2413075696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b400f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2b30009fb0 tx=0x7f2b30009b30).stop 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 -- 172.21.15.112:0/2413075696 shutdown_connections 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 --2- 172.21.15.112:0/2413075696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b400f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 -- 172.21.15.112:0/2413075696 >> 172.21.15.112:0/2413075696 conn(0x7f2b400f1c70 msgr2=0x7f2b400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 -- 172.21.15.112:0/2413075696 shutdown_connections 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.603+0000 7f2b46773700 1 -- 172.21.15.112:0/2413075696 wait complete. 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b46773700 1 Processor -- start 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b46773700 1 -- start start 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b46773700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b46773700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b40106cd0 con 0x7f2b400f6510 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b3ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b3ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34190/0 (socket says 172.21.15.112:34190) 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.604+0000 7f2b3ffff700 1 -- 172.21.15.112:0/1925638084 learned_addr learned my addr 172.21.15.112:0/1925638084 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b3ffff700 1 -- 172.21.15.112:0/1925638084 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b30005040 con 0x7f2b400f6510 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b3ffff700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2b30005010 tx=0x7f2b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b30009320 con 0x7f2b400f6510 2024-02-13T03:34:26.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b40106ed0 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b30013380 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b30014e50 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.605+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b40107360 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b300263a0 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b3cff9700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 0x7f2b2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b40045ad0 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b30040fb0 con 0x7f2b400f6510 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b3f7fe700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 0x7f2b2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.606+0000 7f2b3f7fe700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 0x7f2b2c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2b280096a0 tx=0x7f2b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:26.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.611+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b30012510 con 0x7f2b400f6510 2024-02-13T03:34:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.813+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b40108210 con 0x7f2b400f6510 2024-02-13T03:34:26.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.814+0000 7f2b3cff9700 1 -- 172.21.15.112:0/1925638084 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b30012330 con 0x7f2b400f6510 2024-02-13T03:34:26.815 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:26.815 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 msgr2=0x7f2b2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 0x7f2b2c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2b280096a0 tx=0x7f2b28006b40).stop 2024-02-13T03:34:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 msgr2=0x7f2b401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2b30005010 tx=0x7f2b3000b040).stop 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 shutdown_connections 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f2b2c02ff10 0x7f2b2c0323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 --2- 172.21.15.112:0/1925638084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2b400f6510 0x7f2b401067e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 >> 172.21.15.112:0/1925638084 conn(0x7f2b400f1c70 msgr2=0x7f2b400f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 shutdown_connections 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:26.817+0000 7f2b46773700 1 -- 172.21.15.112:0/1925638084 wait complete. 2024-02-13T03:34:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:24.792985+0000 mgr.smithi080.vyzlfg (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:34:25.980894+0000 mgr.smithi080.vyzlfg (mgr.14162) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi080 ... 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Verifying port 9100 ... 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 1 from systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: systemctl: stderr Job for ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080.service failed because a timeout was exceeded. 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: systemctl: stderr See "systemctl status ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080.service" and "journalctl -xe" for details. 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:34:27.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ports=daemon_ports) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: Failed command: systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-13T03:34:27.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: yield (conn, connr) 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: code, '\n'.join(err))) 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi080 ... 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Verifying port 9100 ... 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 1 from systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: systemctl: stderr Job for ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080.service failed because a timeout was exceeded. 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: systemctl: stderr See "systemctl status ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080.service" and "journalctl -xe" for details. 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-13T03:34:27.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: ports=daemon_ports) 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-13T03:34:27.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: Failed command: systemctl start ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@node-exporter.smithi080 2024-02-13T03:34:27.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:34:25.986317+0000 mgr.smithi080.vyzlfg (mgr.14162) 232 : cephadm [INF] Deploying daemon prometheus.smithi080 on smithi080 2024-02-13T03:34:27.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:27 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:26.815476+0000 mon.smithi080 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.112:0/1925638084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:28.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:28.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:26.793353+0000 mgr.smithi080.vyzlfg (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 -- 172.21.15.112:0/1026453949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 msgr2=0x7fb2000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 --2- 172.21.15.112:0/1026453949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb2000f90f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fb1f0009fb0 tx=0x7fb1f0009b30).stop 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 -- 172.21.15.112:0/1026453949 shutdown_connections 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 --2- 172.21.15.112:0/1026453949 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb2000f90f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 -- 172.21.15.112:0/1026453949 >> 172.21.15.112:0/1026453949 conn(0x7fb2000f0b70 msgr2=0x7fb2000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 -- 172.21.15.112:0/1026453949 shutdown_connections 2024-02-13T03:34:30.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 -- 172.21.15.112:0/1026453949 wait complete. 2024-02-13T03:34:30.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.474+0000 7fb206568700 1 Processor -- start 2024-02-13T03:34:30.475 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb206568700 1 -- start start 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb206568700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb206568700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200106c70 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb1fffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb1fffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34204/0 (socket says 172.21.15.112:34204) 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb1fffff700 1 -- 172.21.15.112:0/684783731 learned_addr learned my addr 172.21.15.112:0/684783731 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.475+0000 7fb1fffff700 1 -- 172.21.15.112:0/684783731 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f0005040 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb1fffff700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb1f00089f0 tx=0x7fb1f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0009320 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb200106e70 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1f0013380 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0014e20 con 0x7fb200060280 2024-02-13T03:34:30.477 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.476+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb200107300 con 0x7fb200060280 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1f00263a0 con 0x7fb200060280 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb1fcff9700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 0x7fb1e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1f0040f00 con 0x7fb200060280 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200045ad0 con 0x7fb200060280 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb1ff7fe700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 0x7fb1e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:30.478 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.477+0000 7fb1ff7fe700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 0x7fb1e80323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb1f40096a0 tx=0x7fb1f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:30.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.482+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1f0012920 con 0x7fb200060280 2024-02-13T03:34:30.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.686+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2001080c0 con 0x7fb200060280 2024-02-13T03:34:30.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.687+0000 7fb1fcff9700 1 -- 172.21.15.112:0/684783731 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1f002b020 con 0x7fb200060280 2024-02-13T03:34:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:30.687 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:30.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 msgr2=0x7fb1e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:30.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 0x7fb1e80323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb1f40096a0 tx=0x7fb1f4006b40).stop 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 msgr2=0x7fb200106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb1f00089f0 tx=0x7fb1f000b040).stop 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 shutdown_connections 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb1e802ff10 0x7fb1e80323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 --2- 172.21.15.112:0/684783731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb200060280 0x7fb200106780 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 >> 172.21.15.112:0/684783731 conn(0x7fb2000f0b70 msgr2=0x7fb2000f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 shutdown_connections 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:30.689+0000 7fb206568700 1 -- 172.21.15.112:0/684783731 wait complete. 2024-02-13T03:34:30.689 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:28.793745+0000 mgr.smithi080.vyzlfg (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:31.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:31 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:30.687661+0000 mon.smithi080 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.112:0/684783731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:32.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:32.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:30.794146+0000 mgr.smithi080.vyzlfg (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- 172.21.15.112:0/1594630436 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 msgr2=0x7fc1180f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 --2- 172.21.15.112:0/1594630436 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1180f9150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fc114009fb0 tx=0x7fc114009b30).stop 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- 172.21.15.112:0/1594630436 shutdown_connections 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 --2- 172.21.15.112:0/1594630436 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1180f9150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- 172.21.15.112:0/1594630436 >> 172.21.15.112:0/1594630436 conn(0x7fc1180f0b70 msgr2=0x7fc1180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- 172.21.15.112:0/1594630436 shutdown_connections 2024-02-13T03:34:34.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- 172.21.15.112:0/1594630436 wait complete. 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 Processor -- start 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.359+0000 7fc120687700 1 -- start start 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc120687700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc120687700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc118106c90 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc11e423700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc11e423700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52826/0 (socket says 172.21.15.112:52826) 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc11e423700 1 -- 172.21.15.112:0/3907847060 learned_addr learned my addr 172.21.15.112:0/3907847060 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc11e423700 1 -- 172.21.15.112:0/3907847060 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114005040 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc11e423700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc1140089f0 tx=0x7fc11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.360+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc114009320 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc114013380 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc118106e90 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc114014e20 con 0x7fc118060220 2024-02-13T03:34:34.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc118107320 con 0x7fc118060220 2024-02-13T03:34:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1140263a0 con 0x7fc118060220 2024-02-13T03:34:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc10effd700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 0x7fc104032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:34.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc114040f60 con 0x7fc118060220 2024-02-13T03:34:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.361+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc118045ad0 con 0x7fc118060220 2024-02-13T03:34:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.362+0000 7fc11dc22700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 0x7fc104032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:34.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.362+0000 7fc11dc22700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 0x7fc104032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:34.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.366+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1140672b0 con 0x7fc118060220 2024-02-13T03:34:34.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.569+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc118108040 con 0x7fc118060220 2024-02-13T03:34:34.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.570+0000 7fc10effd700 1 -- 172.21.15.112:0/3907847060 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc11402b020 con 0x7fc118060220 2024-02-13T03:34:34.570 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:34.570 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 msgr2=0x7fc104032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 0x7fc104032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).stop 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 msgr2=0x7fc1181067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc1140089f0 tx=0x7fc11400b040).stop 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 shutdown_connections 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc10402ff60 0x7fc104032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 --2- 172.21.15.112:0/3907847060 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc118060220 0x7fc1181067a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 >> 172.21.15.112:0/3907847060 conn(0x7fc1180f0b70 msgr2=0x7fc1180f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 shutdown_connections 2024-02-13T03:34:34.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:34.571+0000 7fc120687700 1 -- 172.21.15.112:0/3907847060 wait complete. 2024-02-13T03:34:34.572 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:32.794450+0000 mgr.smithi080.vyzlfg (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:34.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:34.570145+0000 mon.smithi080 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.112:0/3907847060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:36.384 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:36.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:34.794768+0000 mgr.smithi080.vyzlfg (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:38.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2941760066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 msgr2=0x7f8a380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2941760066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a380f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8a20009fb0 tx=0x7f8a20009b30).stop 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2941760066 shutdown_connections 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2941760066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a380f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2941760066 >> 172.21.15.112:0/2941760066 conn(0x7f8a380f0b70 msgr2=0x7f8a380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2941760066 shutdown_connections 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.294+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2941760066 wait complete. 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a3dcff700 1 Processor -- start 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a3dcff700 1 -- start start 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a3dcff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a3dcff700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a38060240 con 0x7f8a380f5410 2024-02-13T03:34:38.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a377fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a377fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52850/0 (socket says 172.21.15.112:52850) 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.295+0000 7f8a377fe700 1 -- 172.21.15.112:0/2483204876 learned_addr learned my addr 172.21.15.112:0/2483204876 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a377fe700 1 -- 172.21.15.112:0/2483204876 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a20005040 con 0x7f8a380f5410 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a377fe700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8a20005010 tx=0x7f8a2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a20009320 con 0x7f8a380f5410 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a38060440 con 0x7f8a380f5410 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a20013380 con 0x7f8a380f5410 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a20014e50 con 0x7f8a380f5410 2024-02-13T03:34:38.298 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.296+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a38109bc0 con 0x7f8a380f5410 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.297+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a200263a0 con 0x7f8a380f5410 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.297+0000 7f8a2ffff700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 0x7f8a240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.297+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a20040fa0 con 0x7f8a380f5410 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.297+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a38045ad0 con 0x7f8a380f5410 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.297+0000 7f8a36ffd700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 0x7f8a240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:38.299 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.298+0000 7f8a36ffd700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 0x7f8a240323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f8a280096a0 tx=0x7f8a28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:38.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.302+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a20012920 con 0x7f8a380f5410 2024-02-13T03:34:38.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.508+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a380fafa0 con 0x7f8a380f5410 2024-02-13T03:34:38.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.509+0000 7f8a2ffff700 1 -- 172.21.15.112:0/2483204876 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a20012330 con 0x7f8a380f5410 2024-02-13T03:34:38.509 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:38.509 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 msgr2=0x7f8a240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 0x7f8a240323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f8a280096a0 tx=0x7f8a28006b40).stop 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 msgr2=0x7f8a38109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8a20005010 tx=0x7f8a2000b040).stop 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 shutdown_connections 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a2402ff10 0x7f8a240323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 --2- 172.21.15.112:0/2483204876 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8a380f5410 0x7f8a38109510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 >> 172.21.15.112:0/2483204876 conn(0x7f8a380f0b70 msgr2=0x7f8a380f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:38.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 shutdown_connections 2024-02-13T03:34:38.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:38.510+0000 7f8a3dcff700 1 -- 172.21.15.112:0/2483204876 wait complete. 2024-02-13T03:34:38.512 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:36.795024+0000 mgr.smithi080.vyzlfg (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:38.509923+0000 mon.smithi080 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.112:0/2483204876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:40.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:40.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:38.795352+0000 mgr.smithi080.vyzlfg (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 -- 172.21.15.112:0/3606425891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 msgr2=0x7f1dbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 --2- 172.21.15.112:0/3606425891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc0f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f1dac009fb0 tx=0x7f1dac009b30).stop 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 -- 172.21.15.112:0/3606425891 shutdown_connections 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 --2- 172.21.15.112:0/3606425891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc0f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 -- 172.21.15.112:0/3606425891 >> 172.21.15.112:0/3606425891 conn(0x7f1dbc0f0b70 msgr2=0x7f1dbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 -- 172.21.15.112:0/3606425891 shutdown_connections 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.074+0000 7f1dc4014700 1 -- 172.21.15.112:0/3606425891 wait complete. 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc4014700 1 Processor -- start 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc4014700 1 -- start start 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc4014700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:42.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc4014700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc105bd0 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc1db0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc1db0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34914/0 (socket says 172.21.15.112:34914) 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.075+0000 7f1dc1db0700 1 -- 172.21.15.112:0/2832839090 learned_addr learned my addr 172.21.15.112:0/2832839090 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1dc1db0700 1 -- 172.21.15.112:0/2832839090 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac005040 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1dc1db0700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1dac005010 tx=0x7f1dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac009320 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1dac013380 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc105dd0 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac014e50 con 0x7f1dbc0f5410 2024-02-13T03:34:42.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.076+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc106260 con 0x7f1dbc0f5410 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.077+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1dac0095f0 con 0x7f1dbc0f5410 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.077+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc045ad0 con 0x7f1dbc0f5410 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.077+0000 7f1db27fc700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 0x7f1da80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.077+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1dac040e90 con 0x7f1dbc0f5410 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.077+0000 7f1dc15af700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 0x7f1da80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:42.079 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.078+0000 7f1dc15af700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 0x7f1da80323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1db80096a0 tx=0x7f1db8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:42.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.082+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dac014420 con 0x7f1dbc0f5410 2024-02-13T03:34:42.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.287+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1dbc107110 con 0x7f1dbc0f5410 2024-02-13T03:34:42.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.288+0000 7f1db27fc700 1 -- 172.21.15.112:0/2832839090 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1dac010230 con 0x7f1dbc0f5410 2024-02-13T03:34:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:42.289 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 msgr2=0x7f1da80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 0x7f1da80323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1db80096a0 tx=0x7f1db8006b40).stop 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 msgr2=0x7f1dbc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1dac005010 tx=0x7f1dac00b040).stop 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 shutdown_connections 2024-02-13T03:34:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1da802ff10 0x7f1da80323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 --2- 172.21.15.112:0/2832839090 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc0f5410 0x7f1dbc1056e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 >> 172.21.15.112:0/2832839090 conn(0x7f1dbc0f0b70 msgr2=0x7f1dbc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.289+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 shutdown_connections 2024-02-13T03:34:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:42.290+0000 7f1dc4014700 1 -- 172.21.15.112:0/2832839090 wait complete. 2024-02-13T03:34:42.291 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:40.795773+0000 mgr.smithi080.vyzlfg (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:42.289279+0000 mon.smithi080 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.112:0/2832839090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:44.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:44.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:42.796167+0000 mgr.smithi080.vyzlfg (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- 172.21.15.112:0/4082178597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 msgr2=0x7ff5d40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 --2- 172.21.15.112:0/4082178597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d40f7d00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff5d0009fb0 tx=0x7ff5d0009b30).stop 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- 172.21.15.112:0/4082178597 shutdown_connections 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 --2- 172.21.15.112:0/4082178597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d40f7d00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- 172.21.15.112:0/4082178597 >> 172.21.15.112:0/4082178597 conn(0x7ff5d40f0120 msgr2=0x7ff5d40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- 172.21.15.112:0/4082178597 shutdown_connections 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- 172.21.15.112:0/4082178597 wait complete. 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 Processor -- start 2024-02-13T03:34:46.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- start start 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.100+0000 7ff5db165700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4106af0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5da163700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5da163700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34938/0 (socket says 172.21.15.112:34938) 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5da163700 1 -- 172.21.15.112:0/3195027206 learned_addr learned my addr 172.21.15.112:0/3195027206 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5da163700 1 -- 172.21.15.112:0/3195027206 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0005040 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5da163700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff5d00089f0 tx=0x7ff5d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0009320 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d4106cf0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5d0013380 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0014e20 con 0x7ff5d40f78e0 2024-02-13T03:34:46.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.101+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d4107180 con 0x7ff5d40f78e0 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.102+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d4045ad0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.102+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5d0014420 con 0x7ff5d40f78e0 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.102+0000 7ff5caffd700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.102+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5d0040ed0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.102+0000 7ff5d9962700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.103+0000 7ff5d9962700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 0x7ff5c00323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff5c40096a0 tx=0x7ff5c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:46.107 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.107+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5d0014790 con 0x7ff5d40f78e0 2024-02-13T03:34:46.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.314+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5d4107fd0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.314+0000 7ff5caffd700 1 -- 172.21.15.112:0/3195027206 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5d001d7f0 con 0x7ff5d40f78e0 2024-02-13T03:34:46.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:46.315 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 msgr2=0x7ff5c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 0x7ff5c00323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff5c40096a0 tx=0x7ff5c4006b40).stop 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 msgr2=0x7ff5d4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff5d00089f0 tx=0x7ff5d000b040).stop 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 shutdown_connections 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff5c002ff10 0x7ff5c00323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 --2- 172.21.15.112:0/3195027206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5d40f78e0 0x7ff5d4106600 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 >> 172.21.15.112:0/3195027206 conn(0x7ff5d40f0120 msgr2=0x7ff5d40f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 shutdown_connections 2024-02-13T03:34:46.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:46.316+0000 7ff5db165700 1 -- 172.21.15.112:0/3195027206 wait complete. 2024-02-13T03:34:46.318 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:44.796499+0000 mgr.smithi080.vyzlfg (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:46.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:46.315693+0000 mon.smithi080 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.112:0/3195027206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:48.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:48.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:46.796843+0000 mgr.smithi080.vyzlfg (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 -- 172.21.15.112:0/1546981210 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 msgr2=0x7fe31c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1546981210 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c0f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fe304009fb0 tx=0x7fe304009b30).stop 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 -- 172.21.15.112:0/1546981210 shutdown_connections 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1546981210 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c0f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 -- 172.21.15.112:0/1546981210 >> 172.21.15.112:0/1546981210 conn(0x7fe31c0f0b70 msgr2=0x7fe31c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:50.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 -- 172.21.15.112:0/1546981210 shutdown_connections 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 -- 172.21.15.112:0/1546981210 wait complete. 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.038+0000 7fe321aa6700 1 Processor -- start 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe321aa6700 1 -- start start 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe321aa6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe321aa6700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c106ce0 con 0x7fe31c0f5410 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe31b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe31b7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34956/0 (socket says 172.21.15.112:34956) 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe31b7fe700 1 -- 172.21.15.112:0/1838475862 learned_addr learned my addr 172.21.15.112:0/1838475862 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe31b7fe700 1 -- 172.21.15.112:0/1838475862 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304005040 con 0x7fe31c0f5410 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.039+0000 7fe31b7fe700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe304005010 tx=0x7fe30400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe304009320 con 0x7fe31c0f5410 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c106ee0 con 0x7fe31c0f5410 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe304013380 con 0x7fe31c0f5410 2024-02-13T03:34:50.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe304014e50 con 0x7fe31c0f5410 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c107370 con 0x7fe31c0f5410 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.040+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3040263a0 con 0x7fe31c0f5410 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.041+0000 7fe313fff700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 0x7fe3080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.041+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe304040e60 con 0x7fe31c0f5410 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.041+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c045ad0 con 0x7fe31c0f5410 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.041+0000 7fe31affd700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 0x7fe3080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.041+0000 7fe31affd700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 0x7fe3080323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe30c0096a0 tx=0x7fe30c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:50.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.045+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe304012920 con 0x7fe31c0f5410 2024-02-13T03:34:50.252 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.252+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe31c1080c0 con 0x7fe31c0f5410 2024-02-13T03:34:50.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.252+0000 7fe313fff700 1 -- 172.21.15.112:0/1838475862 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe30402b020 con 0x7fe31c0f5410 2024-02-13T03:34:50.253 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:50.254 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 msgr2=0x7fe3080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 0x7fe3080323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe30c0096a0 tx=0x7fe30c006b40).stop 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 msgr2=0x7fe31c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe304005010 tx=0x7fe30400b040).stop 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 shutdown_connections 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe30802ff10 0x7fe3080323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 --2- 172.21.15.112:0/1838475862 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe31c0f5410 0x7fe31c1067f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 >> 172.21.15.112:0/1838475862 conn(0x7fe31c0f0b70 msgr2=0x7fe31c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 shutdown_connections 2024-02-13T03:34:50.255 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:50.254+0000 7fe321aa6700 1 -- 172.21.15.112:0/1838475862 wait complete. 2024-02-13T03:34:50.256 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:48.797162+0000 mgr.smithi080.vyzlfg (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:50.253758+0000 mon.smithi080 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.112:0/1838475862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:52.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:52.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:50.797394+0000 mgr.smithi080.vyzlfg (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:54.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- 172.21.15.112:0/612403489 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 msgr2=0x7fd7100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:54.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 --2- 172.21.15.112:0/612403489 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7100f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fd70c009fb0 tx=0x7fd70c009b30).stop 2024-02-13T03:34:54.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- 172.21.15.112:0/612403489 shutdown_connections 2024-02-13T03:34:54.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 --2- 172.21.15.112:0/612403489 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7100f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:54.075 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- 172.21.15.112:0/612403489 >> 172.21.15.112:0/612403489 conn(0x7fd7100f0b70 msgr2=0x7fd7100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- 172.21.15.112:0/612403489 shutdown_connections 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- 172.21.15.112:0/612403489 wait complete. 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 Processor -- start 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.073+0000 7fd717799700 1 -- start start 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd717799700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd717799700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd710108ee0 con 0x7fd7100f5410 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd715535700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:54.076 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd715535700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:58460/0 (socket says 172.21.15.112:58460) 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd715535700 1 -- 172.21.15.112:0/183772107 learned_addr learned my addr 172.21.15.112:0/183772107 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd715535700 1 -- 172.21.15.112:0/183772107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c005040 con 0x7fd7100f5410 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd715535700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd70c005010 tx=0x7fd70c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd70c009320 con 0x7fd7100f5410 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7101090e0 con 0x7fd7100f5410 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd70c013380 con 0x7fd7100f5410 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.074+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd70c014e50 con 0x7fd7100f5410 2024-02-13T03:34:54.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.075+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd710109570 con 0x7fd7100f5410 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.075+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd70c0095f0 con 0x7fd7100f5410 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.075+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710045ad0 con 0x7fd7100f5410 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.076+0000 7fd705ffb700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 0x7fd6fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.076+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd70c040e90 con 0x7fd7100f5410 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.076+0000 7fd714d34700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 0x7fd6fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:54.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.076+0000 7fd714d34700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 0x7fd6fc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd7000096a0 tx=0x7fd700006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:54.081 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.080+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd70c014420 con 0x7fd7100f5410 2024-02-13T03:34:54.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.286+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd71010a420 con 0x7fd7100f5410 2024-02-13T03:34:54.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.287+0000 7fd705ffb700 1 -- 172.21.15.112:0/183772107 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd70c010230 con 0x7fd7100f5410 2024-02-13T03:34:54.288 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:54.288 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:54.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 msgr2=0x7fd6fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:54.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 0x7fd6fc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd7000096a0 tx=0x7fd700006b40).stop 2024-02-13T03:34:54.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 msgr2=0x7fd7101089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:54.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fd70c005010 tx=0x7fd70c00b040).stop 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 shutdown_connections 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd6fc02ff10 0x7fd6fc0323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 --2- 172.21.15.112:0/183772107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd7100f5410 0x7fd7101089f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 >> 172.21.15.112:0/183772107 conn(0x7fd7100f0b70 msgr2=0x7fd7100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 shutdown_connections 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:54.289+0000 7fd717799700 1 -- 172.21.15.112:0/183772107 wait complete. 2024-02-13T03:34:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:52.797724+0000 mgr.smithi080.vyzlfg (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:54.288396+0000 mon.smithi080 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.112:0/183772107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:55.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:55.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:34:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:54.798035+0000 mgr.smithi080.vyzlfg (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:57.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 -- 172.21.15.112:0/431909959 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 msgr2=0x7f7b040fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:57.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/431909959 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b040fb550 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f7af8009fb0 tx=0x7f7af8009b30).stop 2024-02-13T03:34:57.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 -- 172.21.15.112:0/431909959 shutdown_connections 2024-02-13T03:34:57.846 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/431909959 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b040fb550 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 -- 172.21.15.112:0/431909959 >> 172.21.15.112:0/431909959 conn(0x7f7b040f0b70 msgr2=0x7f7b040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 -- 172.21.15.112:0/431909959 shutdown_connections 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 -- 172.21.15.112:0/431909959 wait complete. 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.844+0000 7f7b0ab65700 1 Processor -- start 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b0ab65700 1 -- start start 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b0ab65700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b0ab65700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b04108e90 con 0x7f7b04060220 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b08901700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b08901700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:58466/0 (socket says 172.21.15.112:58466) 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b08901700 1 -- 172.21.15.112:0/2616124597 learned_addr learned my addr 172.21.15.112:0/2616124597 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b08901700 1 -- 172.21.15.112:0/2616124597 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8005040 con 0x7f7b04060220 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.845+0000 7f7b08901700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7af80089f0 tx=0x7f7af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:57.847 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af8009320 con 0x7f7b04060220 2024-02-13T03:34:57.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7af8013380 con 0x7f7b04060220 2024-02-13T03:34:57.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b04109090 con 0x7f7b04060220 2024-02-13T03:34:57.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af8014e20 con 0x7f7b04060220 2024-02-13T03:34:57.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b04109520 con 0x7f7b04060220 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7af8014420 con 0x7f7b04060220 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b017fa700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 0x7f7af40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.846+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b04045ad0 con 0x7f7b04060220 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.847+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7af8040fd0 con 0x7f7b04060220 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.847+0000 7f7b03fff700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 0x7f7af40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:34:57.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.847+0000 7f7b03fff700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 0x7f7af40323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7af00096a0 tx=0x7f7af0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:34:57.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:57.851+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7af801db70 con 0x7f7b04060220 2024-02-13T03:34:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:56.861344+0000 mon.smithi080 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:34:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:56.868730+0000 mon.smithi080 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:34:58.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.053+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b0410a3d0 con 0x7f7b04060220 2024-02-13T03:34:58.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.054+0000 7f7b017fa700 1 -- 172.21.15.112:0/2616124597 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7af801c730 con 0x7f7b04060220 2024-02-13T03:34:58.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:34:58.055 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 msgr2=0x7f7af40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 0x7f7af40323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7af00096a0 tx=0x7f7af0006b40).stop 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 msgr2=0x7f7b041089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f7af80089f0 tx=0x7f7af800b040).stop 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 shutdown_connections 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7af402ff10 0x7f7af40323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 --2- 172.21.15.112:0/2616124597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b04060220 0x7f7b041089a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 >> 172.21.15.112:0/2616124597 conn(0x7f7b040f0b70 msgr2=0x7f7b040f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 shutdown_connections 2024-02-13T03:34:58.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:34:58.056+0000 7f7b0ab65700 1 -- 172.21.15.112:0/2616124597 wait complete. 2024-02-13T03:34:58.058 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:34:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:56.798388+0000 mgr.smithi080.vyzlfg (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:34:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:34:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:34:58.055625+0000 mon.smithi080 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.112:0/2616124597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:34:59.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:34:59.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:34:58.798702+0000 mgr.smithi080.vyzlfg (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:01.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 -- 172.21.15.112:0/212169021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 msgr2=0x7f27a00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:01.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 --2- 172.21.15.112:0/212169021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a00f66c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2790009fb0 tx=0x7f2790009b30).stop 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 -- 172.21.15.112:0/212169021 shutdown_connections 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 --2- 172.21.15.112:0/212169021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a00f66c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 -- 172.21.15.112:0/212169021 >> 172.21.15.112:0/212169021 conn(0x7f27a00f1aa0 msgr2=0x7f27a00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 -- 172.21.15.112:0/212169021 shutdown_connections 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.570+0000 7f27a4955700 1 -- 172.21.15.112:0/212169021 wait complete. 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f27a4955700 1 Processor -- start 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f27a4955700 1 -- start start 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f27a4955700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f27a4955700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0069e20 con 0x7f27a00f62a0 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f279f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f279f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45962/0 (socket says 172.21.15.112:45962) 2024-02-13T03:35:01.585 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.571+0000 7f279f7fe700 1 -- 172.21.15.112:0/4115559102 learned_addr learned my addr 172.21.15.112:0/4115559102 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f279f7fe700 1 -- 172.21.15.112:0/4115559102 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2790005040 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f279f7fe700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f27900089f0 tx=0x7f279000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2790009320 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a0069fc0 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2790013380 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2790014e20 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.572+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a006a450 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.573+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27900095f0 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.573+0000 7f2787fff700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 0x7f27880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.573+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a0059100 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.573+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2790040e90 con 0x7f27a00f62a0 2024-02-13T03:35:01.586 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.573+0000 7f279effd700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 0x7f27880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:01.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.574+0000 7f279effd700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 0x7f27880323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f27940096a0 tx=0x7f2794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:01.587 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.578+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2790014420 con 0x7f27a00f62a0 2024-02-13T03:35:01.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.783+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27a00fbe30 con 0x7f27a00f62a0 2024-02-13T03:35:01.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.785+0000 7f2787fff700 1 -- 172.21.15.112:0/4115559102 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f279002d090 con 0x7f27a00f62a0 2024-02-13T03:35:01.785 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:01.785 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.785+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 msgr2=0x7f27880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.785+0000 7f27a4955700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 0x7f27880323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f27940096a0 tx=0x7f2794006b40).stop 2024-02-13T03:35:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.785+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 msgr2=0x7f27a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.785+0000 7f27a4955700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f27900089f0 tx=0x7f279000b040).stop 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 shutdown_connections 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f278802ff10 0x7f27880323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 --2- 172.21.15.112:0/4115559102 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a00f62a0 0x7f27a006b9a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 >> 172.21.15.112:0/4115559102 conn(0x7f27a00f1aa0 msgr2=0x7f27a00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 shutdown_connections 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:01.786+0000 7f27a4955700 1 -- 172.21.15.112:0/4115559102 wait complete. 2024-02-13T03:35:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:00.799020+0000 mgr.smithi080.vyzlfg (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:01.785333+0000 mon.smithi080 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.112:0/4115559102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:03.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:03.554 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:02.799415+0000 mgr.smithi080.vyzlfg (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:05.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- 172.21.15.112:0/3973826207 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 msgr2=0x7f0b2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:05.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 --2- 172.21.15.112:0/3973826207 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c0f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0b1c009fb0 tx=0x7f0b1c009b30).stop 2024-02-13T03:35:05.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- 172.21.15.112:0/3973826207 shutdown_connections 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 --2- 172.21.15.112:0/3973826207 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c0f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- 172.21.15.112:0/3973826207 >> 172.21.15.112:0/3973826207 conn(0x7f0b2c0f0b70 msgr2=0x7f0b2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- 172.21.15.112:0/3973826207 shutdown_connections 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- 172.21.15.112:0/3973826207 wait complete. 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 Processor -- start 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- start start 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b33efc700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c105bd0 con 0x7f0b2c0f5410 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b31c98700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b31c98700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45974/0 (socket says 172.21.15.112:45974) 2024-02-13T03:35:05.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.273+0000 7f0b31c98700 1 -- 172.21.15.112:0/377191033 learned_addr learned my addr 172.21.15.112:0/377191033 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b31c98700 1 -- 172.21.15.112:0/377191033 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c005040 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b31c98700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0b1c0089f0 tx=0x7f0b1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c009320 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c105dd0 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b1c013380 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1c01ed80 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.275+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c106260 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.276+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c045ad0 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.276+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b1c01e420 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.276+0000 7f0b227fc700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.276+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b1c041060 con 0x7f0b2c0f5410 2024-02-13T03:35:05.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.276+0000 7f0b31497700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:05.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.277+0000 7f0b31497700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 0x7f0b18032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0b280096a0 tx=0x7f0b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:05.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.281+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b1c0095f0 con 0x7f0b2c0f5410 2024-02-13T03:35:05.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.486+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b2c1070b0 con 0x7f0b2c0f5410 2024-02-13T03:35:05.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.487+0000 7f0b227fc700 1 -- 172.21.15.112:0/377191033 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b1c01cb10 con 0x7f0b2c0f5410 2024-02-13T03:35:05.487 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:05.487 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:05.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.488+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 msgr2=0x7f0b18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:05.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.488+0000 7f0b33efc700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 0x7f0b18032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0b280096a0 tx=0x7f0b28006b40).stop 2024-02-13T03:35:05.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 msgr2=0x7f0b2c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:05.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0b1c0089f0 tx=0x7f0b1c00b040).stop 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 shutdown_connections 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 --2- 172.21.15.112:0/377191033 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c1056e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 >> 172.21.15.112:0/377191033 conn(0x7f0b2c0f0b70 msgr2=0x7f0b2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 shutdown_connections 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:05.489+0000 7f0b33efc700 1 -- 172.21.15.112:0/377191033 wait complete. 2024-02-13T03:35:05.489 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:04.799707+0000 mgr.smithi080.vyzlfg (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:05.487621+0000 mon.smithi080 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.112:0/377191033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:07.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:07.160 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:06.800064+0000 mgr.smithi080.vyzlfg (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:09.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 -- 172.21.15.112:0/1320775942 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 msgr2=0x7f78580f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:09.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 --2- 172.21.15.112:0/1320775942 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78580f9150 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f784c009fb0 tx=0x7f784c009b30).stop 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 -- 172.21.15.112:0/1320775942 shutdown_connections 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 --2- 172.21.15.112:0/1320775942 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78580f9150 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 -- 172.21.15.112:0/1320775942 >> 172.21.15.112:0/1320775942 conn(0x7f78580f0b70 msgr2=0x7f78580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 -- 172.21.15.112:0/1320775942 shutdown_connections 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.159+0000 7f785f6d1700 1 -- 172.21.15.112:0/1320775942 wait complete. 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785f6d1700 1 Processor -- start 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785f6d1700 1 -- start start 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785f6d1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:09.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785f6d1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7858106c90 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785d46d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785d46d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45996/0 (socket says 172.21.15.112:45996) 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.160+0000 7f785d46d700 1 -- 172.21.15.112:0/911245847 learned_addr learned my addr 172.21.15.112:0/911245847 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f785d46d700 1 -- 172.21.15.112:0/911245847 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c005040 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f785d46d700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f784c0089f0 tx=0x7f784c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c009320 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f784c013380 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7858106e90 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f784c014e20 con 0x7f7858060220 2024-02-13T03:35:09.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.161+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7858107320 con 0x7f7858060220 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.162+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7858045ad0 con 0x7f7858060220 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.162+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f784c0095f0 con 0x7f7858060220 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.162+0000 7f7849ffb700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 0x7f78440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.162+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f784c040e90 con 0x7f7858060220 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.162+0000 7f785cc6c700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 0x7f78440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:09.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.163+0000 7f785cc6c700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 0x7f78440323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f78540096a0 tx=0x7f7854006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:09.166 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.167+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f784c014420 con 0x7f7858060220 2024-02-13T03:35:09.373 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.374+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7858108160 con 0x7f7858060220 2024-02-13T03:35:09.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.374+0000 7f7849ffb700 1 -- 172.21.15.112:0/911245847 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f784c01d980 con 0x7f7858060220 2024-02-13T03:35:09.374 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:09.374 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 msgr2=0x7f78440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 0x7f78440323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f78540096a0 tx=0x7f7854006b40).stop 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 msgr2=0x7f78581067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f784c0089f0 tx=0x7f784c00b040).stop 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 shutdown_connections 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f784402ff10 0x7f78440323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 --2- 172.21.15.112:0/911245847 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7858060220 0x7f78581067a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 >> 172.21.15.112:0/911245847 conn(0x7f78580f0b70 msgr2=0x7f78580f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 shutdown_connections 2024-02-13T03:35:09.376 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:09.376+0000 7f785f6d1700 1 -- 172.21.15.112:0/911245847 wait complete. 2024-02-13T03:35:09.377 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:09.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:09.374602+0000 mon.smithi080 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.112:0/911245847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:08.800347+0000 mgr.smithi080.vyzlfg (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:11.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:11.067 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:12.830 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:10.800641+0000 mgr.smithi080.vyzlfg (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 -- 172.21.15.112:0/1275148115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 msgr2=0x7f67a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 --2- 172.21.15.112:0/1275148115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a80f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f6798009fb0 tx=0x7f6798009b30).stop 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 -- 172.21.15.112:0/1275148115 shutdown_connections 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 --2- 172.21.15.112:0/1275148115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a80f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 -- 172.21.15.112:0/1275148115 >> 172.21.15.112:0/1275148115 conn(0x7f67a80f0b70 msgr2=0x7f67a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 -- 172.21.15.112:0/1275148115 shutdown_connections 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.920+0000 7f67af6eb700 1 -- 172.21.15.112:0/1275148115 wait complete. 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67af6eb700 1 Processor -- start 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67af6eb700 1 -- start start 2024-02-13T03:35:12.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67af6eb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67af6eb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a8106ca0 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67ad487700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67ad487700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34178/0 (socket says 172.21.15.112:34178) 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.921+0000 7f67ad487700 1 -- 172.21.15.112:0/3845100391 learned_addr learned my addr 172.21.15.112:0/3845100391 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f67ad487700 1 -- 172.21.15.112:0/3845100391 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6798005040 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f67ad487700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6798005010 tx=0x7f679800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6798009320 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a8106ea0 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6798013380 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6798014e50 con 0x7f67a80f5410 2024-02-13T03:35:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.922+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a8107330 con 0x7f67a80f5410 2024-02-13T03:35:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.923+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67980263a0 con 0x7f67a80f5410 2024-02-13T03:35:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.923+0000 7f679dffb700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 0x7f67940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.923+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6798041180 con 0x7f67a80f5410 2024-02-13T03:35:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.923+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a8045ad0 con 0x7f67a80f5410 2024-02-13T03:35:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.923+0000 7f67acc86700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 0x7f67940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:12.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.924+0000 7f67acc86700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 0x7f67940323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f67a40096a0 tx=0x7f67a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:12.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:12.928+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6798012920 con 0x7f67a80f5410 2024-02-13T03:35:13.132 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.132+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67a80fb090 con 0x7f67a80f5410 2024-02-13T03:35:13.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.133+0000 7f679dffb700 1 -- 172.21.15.112:0/3845100391 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f679802b030 con 0x7f67a80f5410 2024-02-13T03:35:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:13.133 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:13.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 msgr2=0x7f67940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:13.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 0x7f67940323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f67a40096a0 tx=0x7f67a4006b40).stop 2024-02-13T03:35:13.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 msgr2=0x7f67a81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:13.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6798005010 tx=0x7f679800b040).stop 2024-02-13T03:35:13.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 shutdown_connections 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f679402ff10 0x7f67940323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 --2- 172.21.15.112:0/3845100391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67a80f5410 0x7f67a81067b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 >> 172.21.15.112:0/3845100391 conn(0x7f67a80f0b70 msgr2=0x7f67a805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.134+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 shutdown_connections 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:13.135+0000 7f67af6eb700 1 -- 172.21.15.112:0/3845100391 wait complete. 2024-02-13T03:35:13.135 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:13.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:13 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:13.133323+0000 mon.smithi080 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.112:0/3845100391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:14.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:14.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:12.801004+0000 mgr.smithi080.vyzlfg (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3826843459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 msgr2=0x7f5ccc0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3826843459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc0f8f20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5cc8009fb0 tx=0x7f5cc8009b30).stop 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3826843459 shutdown_connections 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3826843459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc0f8f20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3826843459 >> 172.21.15.112:0/3826843459 conn(0x7f5ccc0f09a0 msgr2=0x7f5ccc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3826843459 shutdown_connections 2024-02-13T03:35:16.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3826843459 wait complete. 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.727+0000 7f5cd30a7700 1 Processor -- start 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd30a7700 1 -- start start 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd30a7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd30a7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc106af0 con 0x7f5ccc0601e0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd20a5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd20a5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34192/0 (socket says 172.21.15.112:34192) 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.728+0000 7f5cd20a5700 1 -- 172.21.15.112:0/3780401359 learned_addr learned my addr 172.21.15.112:0/3780401359 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cd20a5700 1 -- 172.21.15.112:0/3780401359 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8005040 con 0x7f5ccc0601e0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cd20a5700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5cc8005010 tx=0x7f5cc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc8009320 con 0x7f5ccc0601e0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc106cf0 con 0x7f5ccc0601e0 2024-02-13T03:35:16.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cc8013380 con 0x7f5ccc0601e0 2024-02-13T03:35:16.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc8014e50 con 0x7f5ccc0601e0 2024-02-13T03:35:16.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.729+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc107180 con 0x7f5ccc0601e0 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc045ad0 con 0x7f5ccc0601e0 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cc8014420 con 0x7f5ccc0601e0 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cc2ffd700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cc8040e30 con 0x7f5ccc0601e0 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cd18a4700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.730+0000 7f5cd18a4700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 0x7f5cb80323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5cbc0096a0 tx=0x7f5cbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:16.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.735+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cc80095f0 con 0x7f5ccc0601e0 2024-02-13T03:35:16.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.938+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ccc108030 con 0x7f5ccc0601e0 2024-02-13T03:35:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.939+0000 7f5cc2ffd700 1 -- 172.21.15.112:0/3780401359 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cc801db70 con 0x7f5ccc0601e0 2024-02-13T03:35:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:16.939 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:14.801360+0000 mgr.smithi080.vyzlfg (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 msgr2=0x7f5cb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 0x7f5cb80323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5cbc0096a0 tx=0x7f5cbc006b40).stop 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 msgr2=0x7f5ccc106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5cc8005010 tx=0x7f5cc800b040).stop 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 shutdown_connections 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 --2- 172.21.15.112:0/3780401359 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5ccc0601e0 0x7f5ccc106600 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 >> 172.21.15.112:0/3780401359 conn(0x7f5ccc0f09a0 msgr2=0x7f5ccc0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 shutdown_connections 2024-02-13T03:35:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:16.940+0000 7f5cd30a7700 1 -- 172.21.15.112:0/3780401359 wait complete. 2024-02-13T03:35:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:17.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:17 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:16.940019+0000 mon.smithi080 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.112:0/3780401359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:18.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:18.597 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:16.801618+0000 mgr.smithi080.vyzlfg (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 -- 172.21.15.112:0/2926435581 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 msgr2=0x7f81500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 --2- 172.21.15.112:0/2926435581 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f81500f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f8144009fb0 tx=0x7f8144009b30).stop 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 -- 172.21.15.112:0/2926435581 shutdown_connections 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 --2- 172.21.15.112:0/2926435581 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f81500f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 -- 172.21.15.112:0/2926435581 >> 172.21.15.112:0/2926435581 conn(0x7f81500f1c70 msgr2=0x7f81500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 -- 172.21.15.112:0/2926435581 shutdown_connections 2024-02-13T03:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.602+0000 7f81548e5700 1 -- 172.21.15.112:0/2926435581 wait complete. 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f81548e5700 1 Processor -- start 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f81548e5700 1 -- start start 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f81548e5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f81548e5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150069e20 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f814e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f814e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34208/0 (socket says 172.21.15.112:34208) 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f814e59c700 1 -- 172.21.15.112:0/1616745342 learned_addr learned my addr 172.21.15.112:0/1616745342 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f814e59c700 1 -- 172.21.15.112:0/1616745342 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144005040 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.603+0000 7f814e59c700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f8144005010 tx=0x7f814400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.604+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8144009320 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.604+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8150069fc0 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.604+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8144013380 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.604+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8144014e50 con 0x7f81500f6510 2024-02-13T03:35:20.606 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.604+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815006a450 con 0x7f81500f6510 2024-02-13T03:35:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81440095f0 con 0x7f81500f6510 2024-02-13T03:35:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8150059100 con 0x7f81500f6510 2024-02-13T03:35:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f813effd700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 0x7f81380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8144040e90 con 0x7f81500f6510 2024-02-13T03:35:20.607 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f814dd9b700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 0x7f81380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:20.608 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.605+0000 7f814dd9b700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 0x7f81380323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f81400096a0 tx=0x7f8140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:20.610 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.610+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8144014420 con 0x7f81500f6510 2024-02-13T03:35:20.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.815+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81500fc0a0 con 0x7f81500f6510 2024-02-13T03:35:20.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.816+0000 7f813effd700 1 -- 172.21.15.112:0/1616745342 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8144010230 con 0x7f81500f6510 2024-02-13T03:35:20.817 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:20.817 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:20.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 msgr2=0x7f81380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 0x7f81380323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f81400096a0 tx=0x7f8140006b40).stop 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 msgr2=0x7f815006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f8144005010 tx=0x7f814400b040).stop 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 shutdown_connections 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f813802ff10 0x7f81380323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 --2- 172.21.15.112:0/1616745342 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f81500f6510 0x7f815006b9a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 >> 172.21.15.112:0/1616745342 conn(0x7f81500f1c70 msgr2=0x7f81500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 shutdown_connections 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:20.818+0000 7f81548e5700 1 -- 172.21.15.112:0/1616745342 wait complete. 2024-02-13T03:35:20.819 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:20.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:18.801917+0000 mgr.smithi080.vyzlfg (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:21.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:21 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:20.817186+0000 mon.smithi080 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.112:0/1616745342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:22.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:22.555 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:20.802318+0000 mgr.smithi080.vyzlfg (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.519+0000 7f4743149700 1 -- 172.21.15.112:0/1917673769 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 msgr2=0x7f473c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 --2- 172.21.15.112:0/1917673769 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c0f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f4730009fb0 tx=0x7f4730009b30).stop 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- 172.21.15.112:0/1917673769 shutdown_connections 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 --2- 172.21.15.112:0/1917673769 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c0f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- 172.21.15.112:0/1917673769 >> 172.21.15.112:0/1917673769 conn(0x7f473c0f0b70 msgr2=0x7f473c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- 172.21.15.112:0/1917673769 shutdown_connections 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- 172.21.15.112:0/1917673769 wait complete. 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 Processor -- start 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- start start 2024-02-13T03:35:24.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.520+0000 7f4743149700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c060220 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4740ee5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4740ee5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52766/0 (socket says 172.21.15.112:52766) 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4740ee5700 1 -- 172.21.15.112:0/1671923777 learned_addr learned my addr 172.21.15.112:0/1671923777 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4740ee5700 1 -- 172.21.15.112:0/1671923777 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730005040 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4740ee5700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f47300089f0 tx=0x7f473000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4730009320 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4730013380 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c060420 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f473001ed80 con 0x7f473c0f5410 2024-02-13T03:35:24.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.521+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c109bb0 con 0x7f473c0f5410 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.522+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f473001e420 con 0x7f473c0f5410 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.522+0000 7f47397fa700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 0x7f472c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.522+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c059100 con 0x7f473c0f5410 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.522+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4730041130 con 0x7f473c0f5410 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.522+0000 7f473bfff700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 0x7f472c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:24.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.523+0000 7f473bfff700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 0x7f472c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f47280096a0 tx=0x7f4728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:24.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.527+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47300095f0 con 0x7f473c0f5410 2024-02-13T03:35:24.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.733+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f473c0fafa0 con 0x7f473c0f5410 2024-02-13T03:35:24.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.734+0000 7f47397fa700 1 -- 172.21.15.112:0/1671923777 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4730004490 con 0x7f473c0f5410 2024-02-13T03:35:24.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:24.735 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 msgr2=0x7f472c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 0x7f472c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f47280096a0 tx=0x7f4728006b40).stop 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 msgr2=0x7f473c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f47300089f0 tx=0x7f473000b040).stop 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 shutdown_connections 2024-02-13T03:35:24.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f472c02ff60 0x7f472c032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 --2- 172.21.15.112:0/1671923777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f473c0f5410 0x7f473c109500 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 >> 172.21.15.112:0/1671923777 conn(0x7f473c0f0b70 msgr2=0x7f473c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 shutdown_connections 2024-02-13T03:35:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:24.735+0000 7f4743149700 1 -- 172.21.15.112:0/1671923777 wait complete. 2024-02-13T03:35:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:22.802664+0000 mgr.smithi080.vyzlfg (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:25.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:25 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:24.735256+0000 mon.smithi080 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.112:0/1671923777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:26.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:26.442 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:24.803046+0000 mgr.smithi080.vyzlfg (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:28.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 -- 172.21.15.112:0/1402836827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 msgr2=0x7fbe000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:28.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 --2- 172.21.15.112:0/1402836827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe000f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fbdf0009fb0 tx=0x7fbdf0009b30).stop 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 -- 172.21.15.112:0/1402836827 shutdown_connections 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 --2- 172.21.15.112:0/1402836827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe000f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 -- 172.21.15.112:0/1402836827 >> 172.21.15.112:0/1402836827 conn(0x7fbe000f0b70 msgr2=0x7fbe000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 -- 172.21.15.112:0/1402836827 shutdown_connections 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.279+0000 7fbe0854a700 1 -- 172.21.15.112:0/1402836827 wait complete. 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe0854a700 1 Processor -- start 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe0854a700 1 -- start start 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe0854a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe0854a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00060220 con 0x7fbe000f5410 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe062e6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe062e6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52792/0 (socket says 172.21.15.112:52792) 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe062e6700 1 -- 172.21.15.112:0/2354371943 learned_addr learned my addr 172.21.15.112:0/2354371943 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.280+0000 7fbe062e6700 1 -- 172.21.15.112:0/2354371943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0005040 con 0x7fbe000f5410 2024-02-13T03:35:28.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbe062e6700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fbdf0005010 tx=0x7fbdf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf0009320 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe00060420 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdf0013380 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf0014e50 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.281+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe00109bb0 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbdf0014420 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00059100 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbdf6ffd700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 0x7fbdec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbdf0040e60 con 0x7fbe000f5410 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbe05ae5700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 0x7fbdec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.282+0000 7fbe05ae5700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 0x7fbdec0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbdfc0096a0 tx=0x7fbdfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:28.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.287+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdf00095f0 con 0x7fbe000f5410 2024-02-13T03:35:28.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.495+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe000fafa0 con 0x7fbe000f5410 2024-02-13T03:35:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.496+0000 7fbdf6ffd700 1 -- 172.21.15.112:0/2354371943 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbdf001db70 con 0x7fbe000f5410 2024-02-13T03:35:28.496 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:28.496 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 msgr2=0x7fbdec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 0x7fbdec0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbdfc0096a0 tx=0x7fbdfc006b40).stop 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 msgr2=0x7fbe00109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fbdf0005010 tx=0x7fbdf000b040).stop 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 shutdown_connections 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbdec02ff10 0x7fbdec0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 --2- 172.21.15.112:0/2354371943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe000f5410 0x7fbe00109500 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 >> 172.21.15.112:0/2354371943 conn(0x7fbe000f0b70 msgr2=0x7fbe000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:28.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 shutdown_connections 2024-02-13T03:35:28.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:28.497+0000 7fbe0854a700 1 -- 172.21.15.112:0/2354371943 wait complete. 2024-02-13T03:35:28.499 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:26.803343+0000 mgr.smithi080.vyzlfg (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:29.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:29 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:28.496925+0000 mon.smithi080 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.112:0/2354371943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:30.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:30.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:28.803656+0000 mgr.smithi080.vyzlfg (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/220556496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 msgr2=0x7f5e280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/220556496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e280f90f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f5e1c009fb0 tx=0x7f5e1c009b30).stop 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/220556496 shutdown_connections 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/220556496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e280f90f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/220556496 >> 172.21.15.112:0/220556496 conn(0x7f5e280f0b70 msgr2=0x7f5e280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/220556496 shutdown_connections 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.337+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/220556496 wait complete. 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e2e1f4700 1 Processor -- start 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e2e1f4700 1 -- start start 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e2e1f4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e2e1f4700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28106c70 con 0x7f5e28060280 2024-02-13T03:35:32.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e277fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:32.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e277fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:48106/0 (socket says 172.21.15.112:48106) 2024-02-13T03:35:32.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.338+0000 7f5e277fe700 1 -- 172.21.15.112:0/1478876705 learned_addr learned my addr 172.21.15.112:0/1478876705 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e277fe700 1 -- 172.21.15.112:0/1478876705 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c005040 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e277fe700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5e1c0089f0 tx=0x7f5e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e1c009320 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e28106e70 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e1c013380 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e1c014e20 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.339+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e28107300 con 0x7f5e28060280 2024-02-13T03:35:32.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.340+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e1c0095f0 con 0x7f5e28060280 2024-02-13T03:35:32.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.340+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e28045ad0 con 0x7f5e28060280 2024-02-13T03:35:32.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.340+0000 7f5e0ffff700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:32.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.340+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e1c040ec0 con 0x7f5e28060280 2024-02-13T03:35:32.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.340+0000 7f5e26ffd700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:32.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.341+0000 7f5e26ffd700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 0x7f5e100323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5e180096a0 tx=0x7f5e18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:32.346 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.345+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e1c014420 con 0x7f5e28060280 2024-02-13T03:35:32.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.551+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e28108150 con 0x7f5e28060280 2024-02-13T03:35:32.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.552+0000 7f5e0ffff700 1 -- 172.21.15.112:0/1478876705 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e1c010230 con 0x7f5e28060280 2024-02-13T03:35:32.553 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:32.553 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 msgr2=0x7f5e100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 0x7f5e100323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5e180096a0 tx=0x7f5e18006b40).stop 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 msgr2=0x7f5e28106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f5e1c0089f0 tx=0x7f5e1c00b040).stop 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 shutdown_connections 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 --2- 172.21.15.112:0/1478876705 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e28060280 0x7f5e28106780 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:32.554 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 >> 172.21.15.112:0/1478876705 conn(0x7f5e280f0b70 msgr2=0x7f5e280f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:32.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 shutdown_connections 2024-02-13T03:35:32.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:32.553+0000 7f5e2e1f4700 1 -- 172.21.15.112:0/1478876705 wait complete. 2024-02-13T03:35:32.555 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:30.804061+0000 mgr.smithi080.vyzlfg (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:33.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:33 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:32.553135+0000 mon.smithi080 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.112:0/1478876705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:34.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:34.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:32.804402+0000 mgr.smithi080.vyzlfg (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:36.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 -- 172.21.15.112:0/1213105638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 msgr2=0x7f5e480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:36.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 --2- 172.21.15.112:0/1213105638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f5e38009fb0 tx=0x7f5e38009b30).stop 2024-02-13T03:35:36.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 -- 172.21.15.112:0/1213105638 shutdown_connections 2024-02-13T03:35:36.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 --2- 172.21.15.112:0/1213105638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 -- 172.21.15.112:0/1213105638 >> 172.21.15.112:0/1213105638 conn(0x7f5e480f0b70 msgr2=0x7f5e480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 -- 172.21.15.112:0/1213105638 shutdown_connections 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 -- 172.21.15.112:0/1213105638 wait complete. 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.150+0000 7f5e5037c700 1 Processor -- start 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e5037c700 1 -- start start 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e5037c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e5037c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480fb960 con 0x7f5e480f5410 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e4e118700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e4e118700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:48130/0 (socket says 172.21.15.112:48130) 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e4e118700 1 -- 172.21.15.112:0/3655008639 learned_addr learned my addr 172.21.15.112:0/3655008639 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e4e118700 1 -- 172.21.15.112:0/3655008639 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38005040 con 0x7f5e480f5410 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.151+0000 7f5e4e118700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5e38005010 tx=0x7f5e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:36.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e38009320 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e480fbb00 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e38013380 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e38014e50 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e480fbf90 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e380263a0 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48059100 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.152+0000 7f5e3effd700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 0x7f5e340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.153+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e38041140 con 0x7f5e480f5410 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.153+0000 7f5e4d917700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 0x7f5e340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.153+0000 7f5e4d917700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 0x7f5e340323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5e440096a0 tx=0x7f5e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:36.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.157+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e3801cb10 con 0x7f5e480f5410 2024-02-13T03:35:36.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.360+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e480fafa0 con 0x7f5e480f5410 2024-02-13T03:35:36.361 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.360+0000 7f5e3effd700 1 -- 172.21.15.112:0/3655008639 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e3801cb10 con 0x7f5e480f5410 2024-02-13T03:35:36.361 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:36.361 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 msgr2=0x7f5e340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 0x7f5e340323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5e440096a0 tx=0x7f5e44006b40).stop 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 msgr2=0x7f5e480fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5e38005010 tx=0x7f5e3800b040).stop 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 shutdown_connections 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f5e3402ff10 0x7f5e340323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 --2- 172.21.15.112:0/3655008639 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5e480f5410 0x7f5e480fd4e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 >> 172.21.15.112:0/3655008639 conn(0x7f5e480f0b70 msgr2=0x7f5e4805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 shutdown_connections 2024-02-13T03:35:36.363 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:36.362+0000 7f5e5037c700 1 -- 172.21.15.112:0/3655008639 wait complete. 2024-02-13T03:35:36.364 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:34.804737+0000 mgr.smithi080.vyzlfg (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:36.361806+0000 mon.smithi080 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.112:0/3655008639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:38.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:38.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:36.805073+0000 mgr.smithi080.vyzlfg (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:39.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 -- 172.21.15.112:0/2109544724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 msgr2=0x7fed8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:39.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 --2- 172.21.15.112:0/2109544724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c0f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fed80009fb0 tx=0x7fed80009b30).stop 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 -- 172.21.15.112:0/2109544724 shutdown_connections 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 --2- 172.21.15.112:0/2109544724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c0f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 -- 172.21.15.112:0/2109544724 >> 172.21.15.112:0/2109544724 conn(0x7fed8c0f0b70 msgr2=0x7fed8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 -- 172.21.15.112:0/2109544724 shutdown_connections 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.941+0000 7fed90cc1700 1 -- 172.21.15.112:0/2109544724 wait complete. 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed90cc1700 1 Processor -- start 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed90cc1700 1 -- start start 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed90cc1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed90cc1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c060220 con 0x7fed8c0f5410 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:48152/0 (socket says 172.21.15.112:48152) 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed8a59c700 1 -- 172.21.15.112:0/1135530543 learned_addr learned my addr 172.21.15.112:0/1135530543 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed8a59c700 1 -- 172.21.15.112:0/1135530543 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80005040 con 0x7fed8c0f5410 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.942+0000 7fed8a59c700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fed80005010 tx=0x7fed8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:39.965 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed80009320 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c060420 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed80013380 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed80014e50 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c109bb0 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed800095f0 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.943+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c059100 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.944+0000 7fed7affd700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 0x7fed740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.944+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed80040e90 con 0x7fed8c0f5410 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.944+0000 7fed89d9b700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 0x7fed740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.944+0000 7fed89d9b700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 0x7fed740323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fed7c0096a0 tx=0x7fed7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:39.966 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:39.949+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed80014420 con 0x7fed8c0f5410 2024-02-13T03:35:40.155 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.155+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed8c045ad0 con 0x7fed8c0f5410 2024-02-13T03:35:40.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.156+0000 7fed7affd700 1 -- 172.21.15.112:0/1135530543 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed80010230 con 0x7fed8c0f5410 2024-02-13T03:35:40.156 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:40.156 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:40.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 msgr2=0x7fed740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:40.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 0x7fed740323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fed7c0096a0 tx=0x7fed7c006b40).stop 2024-02-13T03:35:40.158 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 msgr2=0x7fed8c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fed80005010 tx=0x7fed8000b040).stop 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 shutdown_connections 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fed7402ff10 0x7fed740323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 --2- 172.21.15.112:0/1135530543 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fed8c0f5410 0x7fed8c109500 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 >> 172.21.15.112:0/1135530543 conn(0x7fed8c0f0b70 msgr2=0x7fed8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 shutdown_connections 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:40.157+0000 7fed90cc1700 1 -- 172.21.15.112:0/1135530543 wait complete. 2024-02-13T03:35:40.159 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:38.805371+0000 mgr.smithi080.vyzlfg (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:40.157035+0000 mon.smithi080 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.112:0/1135530543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:41.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:41.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:40.805694+0000 mgr.smithi080.vyzlfg (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:43.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/2441659712 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 msgr2=0x7f6a080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:43.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/2441659712 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f69fc009fb0 tx=0x7f69fc009b30).stop 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/2441659712 shutdown_connections 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/2441659712 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/2441659712 >> 172.21.15.112:0/2441659712 conn(0x7f6a080f0b70 msgr2=0x7f6a080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/2441659712 shutdown_connections 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.930+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/2441659712 wait complete. 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0ee8e700 1 Processor -- start 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0ee8e700 1 -- start start 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0ee8e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0ee8e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a080fb960 con 0x7f6a080f5410 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0cc2a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0cc2a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:48316/0 (socket says 172.21.15.112:48316) 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.931+0000 7f6a0cc2a700 1 -- 172.21.15.112:0/1809813549 learned_addr learned my addr 172.21.15.112:0/1809813549 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a0cc2a700 1 -- 172.21.15.112:0/1809813549 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc005040 con 0x7f6a080f5410 2024-02-13T03:35:43.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a0cc2a700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f69fc005010 tx=0x7f69fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc009320 con 0x7f6a080f5410 2024-02-13T03:35:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a080fbb00 con 0x7f6a080f5410 2024-02-13T03:35:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69fc013380 con 0x7f6a080f5410 2024-02-13T03:35:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc014e50 con 0x7f6a080f5410 2024-02-13T03:35:43.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.932+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a080fbf90 con 0x7f6a080f5410 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.933+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69fc0095f0 con 0x7f6a080f5410 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.933+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a08059100 con 0x7f6a080f5410 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.933+0000 7f6a057fa700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 0x7f69f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.933+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69fc040e90 con 0x7f6a080f5410 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.933+0000 7f6a07fff700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 0x7f69f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:43.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.934+0000 7f6a07fff700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 0x7f69f80323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f69f40096a0 tx=0x7f69f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:43.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:43.938+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69fc014420 con 0x7f6a080f5410 2024-02-13T03:35:44.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.142+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a080fafa0 con 0x7f6a080f5410 2024-02-13T03:35:44.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.143+0000 7f6a057fa700 1 -- 172.21.15.112:0/1809813549 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69fc010230 con 0x7f6a080f5410 2024-02-13T03:35:44.142 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:44.142 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:44.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 msgr2=0x7f69f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:44.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 0x7f69f80323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f69f40096a0 tx=0x7f69f4006b40).stop 2024-02-13T03:35:44.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 msgr2=0x7f6a080fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f69fc005010 tx=0x7f69fc00b040).stop 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 shutdown_connections 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f69f802ff10 0x7f69f80323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 --2- 172.21.15.112:0/1809813549 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6a080f5410 0x7f6a080fd4e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 >> 172.21.15.112:0/1809813549 conn(0x7f6a080f0b70 msgr2=0x7f6a0805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 shutdown_connections 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:44.144+0000 7f6a0ee8e700 1 -- 172.21.15.112:0/1809813549 wait complete. 2024-02-13T03:35:44.145 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:42.806008+0000 mgr.smithi080.vyzlfg (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:44.143045+0000 mon.smithi080 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.112:0/1809813549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:45.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:45.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:46.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:44.806288+0000 mgr.smithi080.vyzlfg (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1669928423 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 msgr2=0x7f7b3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1669928423 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c0f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f7b30009fb0 tx=0x7f7b30009b30).stop 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1669928423 shutdown_connections 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1669928423 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c0f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1669928423 >> 172.21.15.112:0/1669928423 conn(0x7f7b3c0f0b70 msgr2=0x7f7b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1669928423 shutdown_connections 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1669928423 wait complete. 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 Processor -- start 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- start start 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:47.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.738+0000 7f7b40ca7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c105c10 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b3a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b3a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:48346/0 (socket says 172.21.15.112:48346) 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b3a59c700 1 -- 172.21.15.112:0/1864874139 learned_addr learned my addr 172.21.15.112:0/1864874139 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b3a59c700 1 -- 172.21.15.112:0/1864874139 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b30005040 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b3a59c700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7b300089f0 tx=0x7f7b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b30009320 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c105e10 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b30013380 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b3001ed80 con 0x7f7b3c0f5410 2024-02-13T03:35:47.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.739+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1062a0 con 0x7f7b3c0f5410 2024-02-13T03:35:47.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.740+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b3001e420 con 0x7f7b3c0f5410 2024-02-13T03:35:47.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.740+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c045ad0 con 0x7f7b3c0f5410 2024-02-13T03:35:47.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.740+0000 7f7b2affd700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 0x7f7b24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:47.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.740+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b30040ea0 con 0x7f7b3c0f5410 2024-02-13T03:35:47.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.740+0000 7f7b39d9b700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 0x7f7b24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:47.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.741+0000 7f7b39d9b700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 0x7f7b24032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7b2c0096a0 tx=0x7f7b2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:47.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.745+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b3001c790 con 0x7f7b3c0f5410 2024-02-13T03:35:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.949+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b3c107150 con 0x7f7b3c0f5410 2024-02-13T03:35:47.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.949+0000 7f7b2affd700 1 -- 172.21.15.112:0/1864874139 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b3001c790 con 0x7f7b3c0f5410 2024-02-13T03:35:47.949 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:47.949 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 msgr2=0x7f7b24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 0x7f7b24032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7b2c0096a0 tx=0x7f7b2c006b40).stop 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 msgr2=0x7f7b3c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7b300089f0 tx=0x7f7b3000b040).stop 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 shutdown_connections 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7b2402ff60 0x7f7b24032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 --2- 172.21.15.112:0/1864874139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b3c0f5410 0x7f7b3c105720 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 >> 172.21.15.112:0/1864874139 conn(0x7f7b3c0f0b70 msgr2=0x7f7b3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 shutdown_connections 2024-02-13T03:35:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:47.951+0000 7f7b40ca7700 1 -- 172.21.15.112:0/1864874139 wait complete. 2024-02-13T03:35:47.952 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:46.806626+0000 mgr.smithi080.vyzlfg (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:47.949704+0000 mon.smithi080 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.112:0/1864874139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:49.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:49.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:48.807026+0000 mgr.smithi080.vyzlfg (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:51.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/2389580141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 msgr2=0x7fb5a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:51.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/2389580141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a00f90f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb588009fb0 tx=0x7fb588009b30).stop 2024-02-13T03:35:51.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/2389580141 shutdown_connections 2024-02-13T03:35:51.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/2389580141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a00f90f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:51.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/2389580141 >> 172.21.15.112:0/2389580141 conn(0x7fb5a00f0b70 msgr2=0x7fb5a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/2389580141 shutdown_connections 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.477+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/2389580141 wait complete. 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb5a4b9a700 1 Processor -- start 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb5a4b9a700 1 -- start start 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb5a4b9a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb5a4b9a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0106c70 con 0x7fb5a0060280 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb59e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb59e59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:58406/0 (socket says 172.21.15.112:58406) 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb59e59c700 1 -- 172.21.15.112:0/3335070913 learned_addr learned my addr 172.21.15.112:0/3335070913 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.478+0000 7fb59e59c700 1 -- 172.21.15.112:0/3335070913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588005040 con 0x7fb5a0060280 2024-02-13T03:35:51.495 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb59e59c700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb5880089f0 tx=0x7fb58800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb588009320 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0106e70 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb588013380 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb588014e20 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.479+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a0107300 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5880095f0 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0045ad0 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb596ffd700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 0x7fb58c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb588040e90 con 0x7fb5a0060280 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb59dd9b700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 0x7fb58c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:51.496 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.480+0000 7fb59dd9b700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 0x7fb58c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb5900096a0 tx=0x7fb590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:51.497 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.484+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb588014420 con 0x7fb5a0060280 2024-02-13T03:35:51.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.689+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5a01081b0 con 0x7fb5a0060280 2024-02-13T03:35:51.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.690+0000 7fb596ffd700 1 -- 172.21.15.112:0/3335070913 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb58801d980 con 0x7fb5a0060280 2024-02-13T03:35:51.690 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:51.690 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 msgr2=0x7fb58c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 0x7fb58c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb5900096a0 tx=0x7fb590006b40).stop 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 msgr2=0x7fb5a0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb5880089f0 tx=0x7fb58800b040).stop 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 shutdown_connections 2024-02-13T03:35:51.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb58c02ff10 0x7fb58c0323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:51.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 --2- 172.21.15.112:0/3335070913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5a0060280 0x7fb5a0106780 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:51.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.691+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 >> 172.21.15.112:0/3335070913 conn(0x7fb5a00f0b70 msgr2=0x7fb5a00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:51.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.692+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 shutdown_connections 2024-02-13T03:35:51.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:51.692+0000 7fb5a4b9a700 1 -- 172.21.15.112:0/3335070913 wait complete. 2024-02-13T03:35:51.692 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:50.807372+0000 mgr.smithi080.vyzlfg (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:51.690249+0000 mon.smithi080 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.112:0/3335070913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:53.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:53.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:52.807714+0000 mgr.smithi080.vyzlfg (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:55.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 -- 172.21.15.112:0/3374477605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 msgr2=0x7fbac40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:55.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3374477605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fbab4009fb0 tx=0x7fbab4009b30).stop 2024-02-13T03:35:55.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 -- 172.21.15.112:0/3374477605 shutdown_connections 2024-02-13T03:35:55.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3374477605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:55.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 -- 172.21.15.112:0/3374477605 >> 172.21.15.112:0/3374477605 conn(0x7fbac40f0b70 msgr2=0x7fbac40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 -- 172.21.15.112:0/3374477605 shutdown_connections 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.172+0000 7fbacbd22700 1 -- 172.21.15.112:0/3374477605 wait complete. 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbacbd22700 1 Processor -- start 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbacbd22700 1 -- start start 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbacbd22700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbacbd22700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40fb960 con 0x7fbac40f5410 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbac9abe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbac9abe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:58422/0 (socket says 172.21.15.112:58422) 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbac9abe700 1 -- 172.21.15.112:0/3124026981 learned_addr learned my addr 172.21.15.112:0/3124026981 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.173+0000 7fbac9abe700 1 -- 172.21.15.112:0/3124026981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4005040 con 0x7fbac40f5410 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbac9abe700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbab4005010 tx=0x7fbab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab4009320 con 0x7fbac40f5410 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac40fbb00 con 0x7fbac40f5410 2024-02-13T03:35:55.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbab4013380 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab4014e50 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.174+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac40fbf90 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbab40263a0 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbaba7fc700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 0x7fbab00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4059100 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbab4041180 con 0x7fbac40f5410 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbac92bd700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 0x7fbab00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.175+0000 7fbac92bd700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 0x7fbab00323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbac00096a0 tx=0x7fbac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:55.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.180+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbab4012920 con 0x7fbac40f5410 2024-02-13T03:35:55.384 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.384+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbac4045ad0 con 0x7fbac40f5410 2024-02-13T03:35:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.385+0000 7fbaba7fc700 1 -- 172.21.15.112:0/3124026981 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbab402c090 con 0x7fbac40f5410 2024-02-13T03:35:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:55.385 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 msgr2=0x7fbab00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 0x7fbab00323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbac00096a0 tx=0x7fbac0006b40).stop 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 msgr2=0x7fbac40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbab4005010 tx=0x7fbab400b040).stop 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 shutdown_connections 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbab002ff10 0x7fbab00323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 --2- 172.21.15.112:0/3124026981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbac40f5410 0x7fbac40fd4e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 >> 172.21.15.112:0/3124026981 conn(0x7fbac40f0b70 msgr2=0x7fbac405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:55.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 shutdown_connections 2024-02-13T03:35:55.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:55.386+0000 7fbacbd22700 1 -- 172.21.15.112:0/3124026981 wait complete. 2024-02-13T03:35:55.388 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:55.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:55.385929+0000 mon.smithi080 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.112:0/3124026981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:35:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:54.807947+0000 mgr.smithi080.vyzlfg (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:57.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:35:57.061 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:35:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:56.862693+0000 mon.smithi080 (mon.0) 274 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:35:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:56.869725+0000 mon.smithi080 (mon.0) 275 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:35:58.935 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- 172.21.15.112:0/2897082153 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 msgr2=0x7f742c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 --2- 172.21.15.112:0/2897082153 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c0f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f7414009fb0 tx=0x7f7414009b30).stop 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- 172.21.15.112:0/2897082153 shutdown_connections 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 --2- 172.21.15.112:0/2897082153 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c0f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- 172.21.15.112:0/2897082153 >> 172.21.15.112:0/2897082153 conn(0x7f742c0f1c70 msgr2=0x7f742c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- 172.21.15.112:0/2897082153 shutdown_connections 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- 172.21.15.112:0/2897082153 wait complete. 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 Processor -- start 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.926+0000 7f7430e94700 1 -- start start 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f7430e94700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f7430e94700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c108ef0 con 0x7f742c0f6510 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f742a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f742a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:58440/0 (socket says 172.21.15.112:58440) 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f742a59c700 1 -- 172.21.15.112:0/2382447606 learned_addr learned my addr 172.21.15.112:0/2382447606 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f742a59c700 1 -- 172.21.15.112:0/2382447606 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414005040 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f742a59c700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f74140089f0 tx=0x7f741400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.927+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414009320 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c1090f0 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7414013380 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741401ed80 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c109580 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f741401e420 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c045ad0 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.928+0000 7f7422ffd700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 0x7f7418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.929+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7414041090 con 0x7f742c0f6510 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.929+0000 7f7429d9b700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 0x7f7418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:35:58.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.929+0000 7f7429d9b700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 0x7f7418032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f741c0096a0 tx=0x7f741c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:35:58.938 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:58.933+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74140095f0 con 0x7f742c0f6510 2024-02-13T03:35:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:56.808213+0000 mgr.smithi080.vyzlfg (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:35:59.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.138+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f742c10a430 con 0x7f742c0f6510 2024-02-13T03:35:59.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.139+0000 7f7422ffd700 1 -- 172.21.15.112:0/2382447606 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7414004490 con 0x7f742c0f6510 2024-02-13T03:35:59.140 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:35:59.140 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.140+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 msgr2=0x7f7418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.140+0000 7f7430e94700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 0x7f7418032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f741c0096a0 tx=0x7f741c006b40).stop 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.140+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 msgr2=0x7f742c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.140+0000 7f7430e94700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f74140089f0 tx=0x7f741400b040).stop 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 shutdown_connections 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f741802ff60 0x7f7418032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:59.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 --2- 172.21.15.112:0/2382447606 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f742c0f6510 0x7f742c108a00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:35:59.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 >> 172.21.15.112:0/2382447606 conn(0x7f742c0f1c70 msgr2=0x7f742c0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:35:59.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 shutdown_connections 2024-02-13T03:35:59.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:35:59.141+0000 7f7430e94700 1 -- 172.21.15.112:0/2382447606 wait complete. 2024-02-13T03:35:59.142 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:35:59.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:35:59 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:35:59.140518+0000 mon.smithi080 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.112:0/2382447606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:00.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:00.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:35:58.808471+0000 mgr.smithi080.vyzlfg (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:02.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- 172.21.15.112:0/955935331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 msgr2=0x7f02180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 --2- 172.21.15.112:0/955935331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02180f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0208009fb0 tx=0x7f0208009b30).stop 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- 172.21.15.112:0/955935331 shutdown_connections 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 --2- 172.21.15.112:0/955935331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02180f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- 172.21.15.112:0/955935331 >> 172.21.15.112:0/955935331 conn(0x7f02180f0b70 msgr2=0x7f02180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- 172.21.15.112:0/955935331 shutdown_connections 2024-02-13T03:36:02.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- 172.21.15.112:0/955935331 wait complete. 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 Processor -- start 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.789+0000 7f021d229700 1 -- start start 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f021d229700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f021d229700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0218106cd0 con 0x7f02180f5410 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f0216d9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f0216d9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42980/0 (socket says 172.21.15.112:42980) 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f0216d9d700 1 -- 172.21.15.112:0/2147148920 learned_addr learned my addr 172.21.15.112:0/2147148920 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f0216d9d700 1 -- 172.21.15.112:0/2147148920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0208005040 con 0x7f02180f5410 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f0216d9d700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0208005010 tx=0x7f020800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0208009320 con 0x7f02180f5410 2024-02-13T03:36:02.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0218106ed0 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0208013380 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.790+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0208014e50 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.791+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0218107360 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.791+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0208014420 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.791+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0218045ad0 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.792+0000 7f01ff7fe700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 0x7f02000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.792+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0208040e30 con 0x7f02180f5410 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.792+0000 7f021659c700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 0x7f02000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:02.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.792+0000 7f021659c700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 0x7f02000323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f020c0096a0 tx=0x7f020c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:02.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:02.796+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02080095f0 con 0x7f02180f5410 2024-02-13T03:36:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:00.808828+0000 mgr.smithi080.vyzlfg (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:03.002 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.002+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0218108210 con 0x7f02180f5410 2024-02-13T03:36:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.002+0000 7f01ff7fe700 1 -- 172.21.15.112:0/2147148920 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f020801db70 con 0x7f02180f5410 2024-02-13T03:36:03.003 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:03.003 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 msgr2=0x7f02000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 0x7f02000323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f020c0096a0 tx=0x7f020c006b40).stop 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 msgr2=0x7f02181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0208005010 tx=0x7f020800b040).stop 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 shutdown_connections 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f020002ff10 0x7f02000323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 --2- 172.21.15.112:0/2147148920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f02180f5410 0x7f02181067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 >> 172.21.15.112:0/2147148920 conn(0x7f02180f0b70 msgr2=0x7f02180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 shutdown_connections 2024-02-13T03:36:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:03.004+0000 7f021d229700 1 -- 172.21.15.112:0/2147148920 wait complete. 2024-02-13T03:36:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:03.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:03 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:03.003888+0000 mon.smithi080 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.112:0/2147148920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:04.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:04.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:02.809113+0000 mgr.smithi080.vyzlfg (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 -- 172.21.15.112:0/478268850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 msgr2=0x7f8ffc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 --2- 172.21.15.112:0/478268850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f8fec009fb0 tx=0x7f8fec009b30).stop 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 -- 172.21.15.112:0/478268850 shutdown_connections 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 --2- 172.21.15.112:0/478268850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 -- 172.21.15.112:0/478268850 >> 172.21.15.112:0/478268850 conn(0x7f8ffc0f0b70 msgr2=0x7f8ffc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 -- 172.21.15.112:0/478268850 shutdown_connections 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.527+0000 7f90028f5700 1 -- 172.21.15.112:0/478268850 wait complete. 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f90028f5700 1 Processor -- start 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f90028f5700 1 -- start start 2024-02-13T03:36:06.530 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f90028f5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f90028f5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0fb960 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f8ffbfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f8ffbfff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:42998/0 (socket says 172.21.15.112:42998) 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.528+0000 7f8ffbfff700 1 -- 172.21.15.112:0/3601821497 learned_addr learned my addr 172.21.15.112:0/3601821497 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f8ffbfff700 1 -- 172.21.15.112:0/3601821497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec005040 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f8ffbfff700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f8fec0089f0 tx=0x7f8fec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fec009320 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0fbb00 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fec013380 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fec01ed80 con 0x7f8ffc0f5410 2024-02-13T03:36:06.531 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.529+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc0fbf90 con 0x7f8ffc0f5410 2024-02-13T03:36:06.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.530+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fec014420 con 0x7f8ffc0f5410 2024-02-13T03:36:06.532 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.530+0000 7f8ff8ff9700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 0x7f8fe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:06.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.530+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fec040f00 con 0x7f8ffc0f5410 2024-02-13T03:36:06.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.530+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc059100 con 0x7f8ffc0f5410 2024-02-13T03:36:06.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.530+0000 7f8ffb7fe700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 0x7f8fe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:06.533 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.531+0000 7f8ffb7fe700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 0x7f8fe4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8ff00096a0 tx=0x7f8ff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:06.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.535+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fec01c910 con 0x7f8ffc0f5410 2024-02-13T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.740+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ffc0fc690 con 0x7f8ffc0f5410 2024-02-13T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.741+0000 7f8ff8ff9700 1 -- 172.21.15.112:0/3601821497 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fec01c320 con 0x7f8ffc0f5410 2024-02-13T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 msgr2=0x7f8fe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 0x7f8fe4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8ff00096a0 tx=0x7f8ff0006b40).stop 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 msgr2=0x7f8ffc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f8fec0089f0 tx=0x7f8fec00b040).stop 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 shutdown_connections 2024-02-13T03:36:06.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8fe402ff60 0x7f8fe4032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:06.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 --2- 172.21.15.112:0/3601821497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8ffc0f5410 0x7f8ffc0fd4e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:06.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.742+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 >> 172.21.15.112:0/3601821497 conn(0x7f8ffc0f0b70 msgr2=0x7f8ffc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:06.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.743+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 shutdown_connections 2024-02-13T03:36:06.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:06.743+0000 7f90028f5700 1 -- 172.21.15.112:0/3601821497 wait complete. 2024-02-13T03:36:06.744 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:04.809466+0000 mgr.smithi080.vyzlfg (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:07.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:06.742137+0000 mon.smithi080 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.112:0/3601821497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:08.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:08.443 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:06.809800+0000 mgr.smithi080.vyzlfg (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:10.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 -- 172.21.15.112:0/2176921827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:10.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 --2- 172.21.15.112:0/2176921827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fcb80009fb0 tx=0x7fcb80009b30).stop 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 -- 172.21.15.112:0/2176921827 shutdown_connections 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 --2- 172.21.15.112:0/2176921827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 -- 172.21.15.112:0/2176921827 >> 172.21.15.112:0/2176921827 conn(0x7fcb900f0b70 msgr2=0x7fcb900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 -- 172.21.15.112:0/2176921827 shutdown_connections 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.419+0000 7fcb96510700 1 -- 172.21.15.112:0/2176921827 wait complete. 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb96510700 1 Processor -- start 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb96510700 1 -- start start 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb96510700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb96510700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb90060220 con 0x7fcb900f5410 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb8ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb8ffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43002/0 (socket says 172.21.15.112:43002) 2024-02-13T03:36:10.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.420+0000 7fcb8ffff700 1 -- 172.21.15.112:0/2088480987 learned_addr learned my addr 172.21.15.112:0/2088480987 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb8ffff700 1 -- 172.21.15.112:0/2088480987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb80005040 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb8ffff700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcb80005010 tx=0x7fcb8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb80009320 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb90060420 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb80013380 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb80014e50 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.421+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb90109bb0 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.422+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb80014420 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.422+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb90059100 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.422+0000 7fcb8cff9700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 0x7fcb7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.422+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb80040f40 con 0x7fcb900f5410 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.422+0000 7fcb8f7fe700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 0x7fcb7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.423+0000 7fcb8f7fe700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 0x7fcb7c0323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcb780096a0 tx=0x7fcb78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:10.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.427+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb800095f0 con 0x7fcb900f5410 2024-02-13T03:36:10.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.631+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb90045ad0 con 0x7fcb900f5410 2024-02-13T03:36:10.632 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.632+0000 7fcb8cff9700 1 -- 172.21.15.112:0/2088480987 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb8001db70 con 0x7fcb900f5410 2024-02-13T03:36:10.633 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:10.633 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:10.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 msgr2=0x7fcb7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:10.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 0x7fcb7c0323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcb780096a0 tx=0x7fcb78006b40).stop 2024-02-13T03:36:10.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb90109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcb80005010 tx=0x7fcb8000b040).stop 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 shutdown_connections 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fcb7c02ff10 0x7fcb7c0323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 --2- 172.21.15.112:0/2088480987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcb900f5410 0x7fcb90109500 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 >> 172.21.15.112:0/2088480987 conn(0x7fcb900f0b70 msgr2=0x7fcb900f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 shutdown_connections 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:10.633+0000 7fcb96510700 1 -- 172.21.15.112:0/2088480987 wait complete. 2024-02-13T03:36:10.635 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:08.810154+0000 mgr.smithi080.vyzlfg (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:11.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:11 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:10.633138+0000 mon.smithi080 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.112:0/2088480987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:12.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:12.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:10.810458+0000 mgr.smithi080.vyzlfg (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:14.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- 172.21.15.112:0/1605858101 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 msgr2=0x7fa5f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:14.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/1605858101 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f80f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa5f0009960 tx=0x7fa5f0004d10).stop 2024-02-13T03:36:14.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- 172.21.15.112:0/1605858101 shutdown_connections 2024-02-13T03:36:14.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/1605858101 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f80f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- 172.21.15.112:0/1605858101 >> 172.21.15.112:0/1605858101 conn(0x7fa5f80f1c70 msgr2=0x7fa5f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- 172.21.15.112:0/1605858101 shutdown_connections 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- 172.21.15.112:0/1605858101 wait complete. 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 Processor -- start 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.329+0000 7fa5fed1d700 1 -- start start 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fed1d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fed1d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8106cd0 con 0x7fa5f80f6510 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fcab9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fcab9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:56446/0 (socket says 172.21.15.112:56446) 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fcab9700 1 -- 172.21.15.112:0/2683401061 learned_addr learned my addr 172.21.15.112:0/2683401061 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fcab9700 1 -- 172.21.15.112:0/2683401061 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0009640 con 0x7fa5f80f6510 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.330+0000 7fa5fcab9700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa5f0004b70 tx=0x7fa5f0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:14.332 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f0018070 con 0x7fa5f80f6510 2024-02-13T03:36:14.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5f000b970 con 0x7fa5f80f6510 2024-02-13T03:36:14.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8106ed0 con 0x7fa5f80f6510 2024-02-13T03:36:14.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f001c560 con 0x7fa5f80f6510 2024-02-13T03:36:14.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f8107360 con 0x7fa5f80f6510 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5f0013630 con 0x7fa5f80f6510 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.331+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8045ad0 con 0x7fa5f80f6510 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.332+0000 7fa5f57fa700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 0x7fa5e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.332+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5f0041270 con 0x7fa5f80f6510 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.332+0000 7fa5f7fff700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 0x7fa5e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.332+0000 7fa5f7fff700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 0x7fa5e0032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa5ec0096a0 tx=0x7fa5ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:14.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.337+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5f0013a30 con 0x7fa5f80f6510 2024-02-13T03:36:14.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.540+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5f8108210 con 0x7fa5f80f6510 2024-02-13T03:36:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.541+0000 7fa5f57fa700 1 -- 172.21.15.112:0/2683401061 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5f0012380 con 0x7fa5f80f6510 2024-02-13T03:36:14.541 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:14.541 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 msgr2=0x7fa5e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 0x7fa5e0032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa5ec0096a0 tx=0x7fa5ec006b40).stop 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 msgr2=0x7fa5f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa5f0004b70 tx=0x7fa5f0004d10).stop 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 shutdown_connections 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa5e002ffb0 0x7fa5e0032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 --2- 172.21.15.112:0/2683401061 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa5f80f6510 0x7fa5f81067e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 >> 172.21.15.112:0/2683401061 conn(0x7fa5f80f1c70 msgr2=0x7fa5f80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:14.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 shutdown_connections 2024-02-13T03:36:14.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:14.542+0000 7fa5fed1d700 1 -- 172.21.15.112:0/2683401061 wait complete. 2024-02-13T03:36:14.544 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:12.810722+0000 mgr.smithi080.vyzlfg (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:14.542057+0000 mon.smithi080 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.112:0/2683401061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:16.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:16.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:14.811098+0000 mgr.smithi080.vyzlfg (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:18.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 -- 172.21.15.112:0/95914891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 msgr2=0x7f288c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:18.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 --2- 172.21.15.112:0/95914891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c0f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f287c009fb0 tx=0x7f287c009b30).stop 2024-02-13T03:36:18.300 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 -- 172.21.15.112:0/95914891 shutdown_connections 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 --2- 172.21.15.112:0/95914891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c0f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 -- 172.21.15.112:0/95914891 >> 172.21.15.112:0/95914891 conn(0x7f288c0f09a0 msgr2=0x7f288c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 -- 172.21.15.112:0/95914891 shutdown_connections 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.289+0000 7f288b59e700 1 -- 172.21.15.112:0/95914891 wait complete. 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288b59e700 1 Processor -- start 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288b59e700 1 -- start start 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288b59e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288b59e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1059c0 con 0x7f288c0f51a0 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:56460/0 (socket says 172.21.15.112:56460) 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.290+0000 7f288a59c700 1 -- 172.21.15.112:0/3304723903 learned_addr learned my addr 172.21.15.112:0/3304723903 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f288a59c700 1 -- 172.21.15.112:0/3304723903 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c005040 con 0x7f288c0f51a0 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f288a59c700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f287c0089f0 tx=0x7f287c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f287c009320 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c105bc0 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c106050 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f287c013380 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.291+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f287c014e40 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.292+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f287c014420 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.292+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c045ad0 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.292+0000 7f287affd700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 0x7f28740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.292+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f287c042ff0 con 0x7f288c0f51a0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.292+0000 7f2889d9b700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 0x7f28740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.293+0000 7f2889d9b700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 0x7f28740323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f28800096a0 tx=0x7f2880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:18.302 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.297+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f287c06f2b0 con 0x7f288c0f51a0 2024-02-13T03:36:18.503 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.503+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f288c106f00 con 0x7f288c0f51a0 2024-02-13T03:36:18.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.504+0000 7f287affd700 1 -- 172.21.15.112:0/3304723903 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f287c012980 con 0x7f288c0f51a0 2024-02-13T03:36:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:18.504 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 msgr2=0x7f28740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 0x7f28740323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f28800096a0 tx=0x7f2880006b40).stop 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 msgr2=0x7f288c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f287c0089f0 tx=0x7f287c00b040).stop 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 shutdown_connections 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f287402ff10 0x7f28740323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 --2- 172.21.15.112:0/3304723903 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f288c0f51a0 0x7f288c1054d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 >> 172.21.15.112:0/3304723903 conn(0x7f288c0f09a0 msgr2=0x7f288c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:18.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 shutdown_connections 2024-02-13T03:36:18.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:18.505+0000 7f288b59e700 1 -- 172.21.15.112:0/3304723903 wait complete. 2024-02-13T03:36:18.507 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:16.811463+0000 mgr.smithi080.vyzlfg (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:18.504975+0000 mon.smithi080 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.112:0/3304723903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:20.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:20.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:20.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:18.811838+0000 mgr.smithi080.vyzlfg (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:22.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 -- 172.21.15.112:0/1792400814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 msgr2=0x7f27b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:22.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/1792400814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b40f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f27a4009fb0 tx=0x7f27a4009b30).stop 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 -- 172.21.15.112:0/1792400814 shutdown_connections 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/1792400814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b40f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 -- 172.21.15.112:0/1792400814 >> 172.21.15.112:0/1792400814 conn(0x7f27b40f0b70 msgr2=0x7f27b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 -- 172.21.15.112:0/1792400814 shutdown_connections 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.119+0000 7f27bb6b7700 1 -- 172.21.15.112:0/1792400814 wait complete. 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27bb6b7700 1 Processor -- start 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27bb6b7700 1 -- start start 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27bb6b7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27bb6b7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b4106d00 con 0x7f27b40f5410 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27b9453700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27b9453700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:51694/0 (socket says 172.21.15.112:51694) 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27b9453700 1 -- 172.21.15.112:0/4238619453 learned_addr learned my addr 172.21.15.112:0/4238619453 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27b9453700 1 -- 172.21.15.112:0/4238619453 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4005040 con 0x7f27b40f5410 2024-02-13T03:36:22.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.120+0000 7f27b9453700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f27a4005010 tx=0x7f27a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.121+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a4009320 con 0x7f27b40f5410 2024-02-13T03:36:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.121+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a4013380 con 0x7f27b40f5410 2024-02-13T03:36:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.121+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b4106f00 con 0x7f27b40f5410 2024-02-13T03:36:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.121+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a4014e50 con 0x7f27b40f5410 2024-02-13T03:36:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.121+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b4107390 con 0x7f27b40f5410 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27a40095f0 con 0x7f27b40f5410 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b4045ad0 con 0x7f27b40f5410 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27a9ffb700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 0x7f27a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27a4040e90 con 0x7f27b40f5410 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27b8c52700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 0x7f27a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.122+0000 7f27b8c52700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 0x7f27a00323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f27b00096a0 tx=0x7f27b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:22.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.127+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27a4014420 con 0x7f27b40f5410 2024-02-13T03:36:22.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.331+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27b4108240 con 0x7f27b40f5410 2024-02-13T03:36:22.331 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.332+0000 7f27a9ffb700 1 -- 172.21.15.112:0/4238619453 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27a4010230 con 0x7f27b40f5410 2024-02-13T03:36:22.332 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:22.332 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.333+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 msgr2=0x7f27a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.333+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 0x7f27a00323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f27b00096a0 tx=0x7f27b0006b40).stop 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 msgr2=0x7f27b4106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f27a4005010 tx=0x7f27a400b040).stop 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 shutdown_connections 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f27a002ff10 0x7f27a00323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 --2- 172.21.15.112:0/4238619453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27b40f5410 0x7f27b4106810 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 >> 172.21.15.112:0/4238619453 conn(0x7f27b40f0b70 msgr2=0x7f27b405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 shutdown_connections 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:22.334+0000 7f27bb6b7700 1 -- 172.21.15.112:0/4238619453 wait complete. 2024-02-13T03:36:22.334 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:20.812151+0000 mgr.smithi080.vyzlfg (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:22.332337+0000 mon.smithi080 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.112:0/4238619453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:24.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:24.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:22.812514+0000 mgr.smithi080.vyzlfg (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:26.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 -- 172.21.15.112:0/2161617908 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 msgr2=0x7f59980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:26.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 --2- 172.21.15.112:0/2161617908 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f59980f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f598c009fb0 tx=0x7f598c009b30).stop 2024-02-13T03:36:26.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 -- 172.21.15.112:0/2161617908 shutdown_connections 2024-02-13T03:36:26.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 --2- 172.21.15.112:0/2161617908 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f59980f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:26.025 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 -- 172.21.15.112:0/2161617908 >> 172.21.15.112:0/2161617908 conn(0x7f59980f0b70 msgr2=0x7f59980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 -- 172.21.15.112:0/2161617908 shutdown_connections 2024-02-13T03:36:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.025+0000 7f599eaf9700 1 -- 172.21.15.112:0/2161617908 wait complete. 2024-02-13T03:36:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599eaf9700 1 Processor -- start 2024-02-13T03:36:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599eaf9700 1 -- start start 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599eaf9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599eaf9700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981072f0 con 0x7f59980f5410 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599c895700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599c895700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:51702/0 (socket says 172.21.15.112:51702) 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599c895700 1 -- 172.21.15.112:0/699399029 learned_addr learned my addr 172.21.15.112:0/699399029 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.026+0000 7f599c895700 1 -- 172.21.15.112:0/699399029 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c005040 con 0x7f59980f5410 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f599c895700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f598c005010 tx=0x7f598c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598c009320 con 0x7f59980f5410 2024-02-13T03:36:26.028 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5998060220 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f598c013380 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598c014e50 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599810bdf0 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.027+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f598c014420 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.028+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998045ad0 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.028+0000 7f59957fa700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 0x7f59880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.028+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f598c040e30 con 0x7f59980f5410 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.028+0000 7f5997fff700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 0x7f59880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:26.029 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.028+0000 7f5997fff700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 0x7f59880323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f59840096a0 tx=0x7f5984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:26.032 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.032+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f598c0095f0 con 0x7f59980f5410 2024-02-13T03:36:26.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.235+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f599810c5b0 con 0x7f59980f5410 2024-02-13T03:36:26.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.236+0000 7f59957fa700 1 -- 172.21.15.112:0/699399029 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f598c01db70 con 0x7f59980f5410 2024-02-13T03:36:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:26.236 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:26.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 msgr2=0x7f59880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:26.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 0x7f59880323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f59840096a0 tx=0x7f5984006b40).stop 2024-02-13T03:36:26.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 msgr2=0x7f599810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:26.237 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f598c005010 tx=0x7f598c00b040).stop 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 shutdown_connections 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f598802ff10 0x7f59880323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 --2- 172.21.15.112:0/699399029 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59980f5410 0x7f599810b900 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.237+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 >> 172.21.15.112:0/699399029 conn(0x7f59980f0b70 msgr2=0x7f59980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.238+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 shutdown_connections 2024-02-13T03:36:26.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:26.238+0000 7f599eaf9700 1 -- 172.21.15.112:0/699399029 wait complete. 2024-02-13T03:36:26.239 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:24.812778+0000 mgr.smithi080.vyzlfg (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:26.236338+0000 mon.smithi080 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.112:0/699399029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:27.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:27.980 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:26.813175+0000 mgr.smithi080.vyzlfg (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 -- 172.21.15.112:0/3390668634 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 msgr2=0x7f617c0f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 --2- 172.21.15.112:0/3390668634 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7dc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f6170009fb0 tx=0x7f6170009b30).stop 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 -- 172.21.15.112:0/3390668634 shutdown_connections 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 --2- 172.21.15.112:0/3390668634 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7dc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 -- 172.21.15.112:0/3390668634 >> 172.21.15.112:0/3390668634 conn(0x7f617c0f0120 msgr2=0x7f617c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 -- 172.21.15.112:0/3390668634 shutdown_connections 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.960+0000 7f6182615700 1 -- 172.21.15.112:0/3390668634 wait complete. 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6182615700 1 Processor -- start 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6182615700 1 -- start start 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6182615700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6182615700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0f56f0 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6181613700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6181613700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:51730/0 (socket says 172.21.15.112:51730) 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.961+0000 7f6181613700 1 -- 172.21.15.112:0/250248987 learned_addr learned my addr 172.21.15.112:0/250248987 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f6181613700 1 -- 172.21.15.112:0/250248987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170005040 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f6181613700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f61700089f0 tx=0x7f617000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6170009320 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c0f5890 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6170013380 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6170014e20 con 0x7f617c0f79a0 2024-02-13T03:36:29.963 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.962+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c0f5d20 con 0x7f617c0f79a0 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.963+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c059100 con 0x7f617c0f79a0 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.963+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61700095f0 con 0x7f617c0f79a0 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.963+0000 7f616dffb700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 0x7f61680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.963+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6170040ec0 con 0x7f617c0f79a0 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.963+0000 7f6180e12700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 0x7f61680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:29.964 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.964+0000 7f6180e12700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 0x7f61680323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f61780096a0 tx=0x7f6178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:29.967 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:29.968+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6170014420 con 0x7f617c0f79a0 2024-02-13T03:36:30.172 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.173+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f617c045ad0 con 0x7f617c0f79a0 2024-02-13T03:36:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.173+0000 7f616dffb700 1 -- 172.21.15.112:0/250248987 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6170010230 con 0x7f617c0f79a0 2024-02-13T03:36:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:30.173 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 msgr2=0x7f61680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 0x7f61680323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f61780096a0 tx=0x7f6178006b40).stop 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 msgr2=0x7f617c0f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f61700089f0 tx=0x7f617000b040).stop 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 shutdown_connections 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f616802ff10 0x7f61680323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 --2- 172.21.15.112:0/250248987 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f617c0f79a0 0x7f617c0f7270 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 >> 172.21.15.112:0/250248987 conn(0x7f617c0f0120 msgr2=0x7f617c0f0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 shutdown_connections 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:30.175+0000 7f6182615700 1 -- 172.21.15.112:0/250248987 wait complete. 2024-02-13T03:36:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:28.813506+0000 mgr.smithi080.vyzlfg (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:30.173731+0000 mon.smithi080 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.112:0/250248987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:31.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:31.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:30.813844+0000 mgr.smithi080.vyzlfg (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:33.908 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 -- 172.21.15.112:0/4053110242 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 msgr2=0x7f0eb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 --2- 172.21.15.112:0/4053110242 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb80f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f0ea8009fb0 tx=0x7f0ea8009b30).stop 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 -- 172.21.15.112:0/4053110242 shutdown_connections 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 --2- 172.21.15.112:0/4053110242 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb80f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 -- 172.21.15.112:0/4053110242 >> 172.21.15.112:0/4053110242 conn(0x7f0eb80f1c70 msgr2=0x7f0eb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 -- 172.21.15.112:0/4053110242 shutdown_connections 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 -- 172.21.15.112:0/4053110242 wait complete. 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.906+0000 7f0ebf903700 1 Processor -- start 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebf903700 1 -- start start 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebf903700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebf903700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8106cd0 con 0x7f0eb80f6510 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebd69f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebd69f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34248/0 (socket says 172.21.15.112:34248) 2024-02-13T03:36:33.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebd69f700 1 -- 172.21.15.112:0/67824252 learned_addr learned my addr 172.21.15.112:0/67824252 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.907+0000 7f0ebd69f700 1 -- 172.21.15.112:0/67824252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8005040 con 0x7f0eb80f6510 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0ebd69f700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0ea80089f0 tx=0x7f0ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ea8009320 con 0x7f0eb80f6510 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ea8013380 con 0x7f0eb80f6510 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb8106ed0 con 0x7f0eb80f6510 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ea801ed80 con 0x7f0eb80f6510 2024-02-13T03:36:33.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.908+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb8107360 con 0x7f0eb80f6510 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.909+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ea801e420 con 0x7f0eb80f6510 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.909+0000 7f0eadffb700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 0x7f0ea4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.909+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb8045ad0 con 0x7f0eb80f6510 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.909+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ea8040fc0 con 0x7f0eb80f6510 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.909+0000 7f0ebce9e700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 0x7f0ea4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:33.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.910+0000 7f0ebce9e700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 0x7f0ea4032470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0eb40096a0 tx=0x7f0eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:33.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:33.914+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ea80095f0 con 0x7f0eb80f6510 2024-02-13T03:36:34.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.119+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eb81081b0 con 0x7f0eb80f6510 2024-02-13T03:36:34.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.120+0000 7f0eadffb700 1 -- 172.21.15.112:0/67824252 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ea802c2d0 con 0x7f0eb80f6510 2024-02-13T03:36:34.121 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:34.121 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:34.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 msgr2=0x7f0ea4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 0x7f0ea4032470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0eb40096a0 tx=0x7f0eb4006b40).stop 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 msgr2=0x7f0eb81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0ea80089f0 tx=0x7f0ea800b040).stop 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 shutdown_connections 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0ea402ffb0 0x7f0ea4032470 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 --2- 172.21.15.112:0/67824252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0eb80f6510 0x7f0eb81067e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 >> 172.21.15.112:0/67824252 conn(0x7f0eb80f1c70 msgr2=0x7f0eb80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 shutdown_connections 2024-02-13T03:36:34.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:34.122+0000 7f0ebf903700 1 -- 172.21.15.112:0/67824252 wait complete. 2024-02-13T03:36:34.124 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:32.814188+0000 mgr.smithi080.vyzlfg (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:34.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:34.121300+0000 mon.smithi080 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.112:0/67824252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:35.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:35.772 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:34.814524+0000 mgr.smithi080.vyzlfg (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 -- 172.21.15.112:0/3065806282 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 msgr2=0x7f8b980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 --2- 172.21.15.112:0/3065806282 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b980f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f8b8c009fb0 tx=0x7f8b8c009b30).stop 2024-02-13T03:36:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 -- 172.21.15.112:0/3065806282 shutdown_connections 2024-02-13T03:36:37.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 --2- 172.21.15.112:0/3065806282 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b980f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 -- 172.21.15.112:0/3065806282 >> 172.21.15.112:0/3065806282 conn(0x7f8b980f0b70 msgr2=0x7f8b980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 -- 172.21.15.112:0/3065806282 shutdown_connections 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.763+0000 7f8b9d967700 1 -- 172.21.15.112:0/3065806282 wait complete. 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b9d967700 1 Processor -- start 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b9d967700 1 -- start start 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b9d967700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b9d967700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98106d00 con 0x7f8b980f5410 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b96ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b96ffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:34266/0 (socket says 172.21.15.112:34266) 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b96ffd700 1 -- 172.21.15.112:0/2596156642 learned_addr learned my addr 172.21.15.112:0/2596156642 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b96ffd700 1 -- 172.21.15.112:0/2596156642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c005040 con 0x7f8b980f5410 2024-02-13T03:36:37.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.764+0000 7f8b96ffd700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8b8c005010 tx=0x7f8b8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8c009320 con 0x7f8b980f5410 2024-02-13T03:36:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98106f00 con 0x7f8b980f5410 2024-02-13T03:36:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b8c013380 con 0x7f8b980f5410 2024-02-13T03:36:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8c014e50 con 0x7f8b980f5410 2024-02-13T03:36:37.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98107390 con 0x7f8b980f5410 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.765+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b8c0095f0 con 0x7f8b980f5410 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.766+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b98045ad0 con 0x7f8b980f5410 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.766+0000 7f8b7f7fe700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 0x7f8b800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.766+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b8c040f70 con 0x7f8b980f5410 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.766+0000 7f8b967fc700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 0x7f8b800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:37.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.766+0000 7f8b967fc700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 0x7f8b800323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8b880096a0 tx=0x7f8b88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:37.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.771+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b8c014420 con 0x7f8b980f5410 2024-02-13T03:36:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.974+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b980fafa0 con 0x7f8b980f5410 2024-02-13T03:36:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.975+0000 7f8b7f7fe700 1 -- 172.21.15.112:0/2596156642 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b8c01d980 con 0x7f8b980f5410 2024-02-13T03:36:37.975 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:37.976 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 msgr2=0x7f8b800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 0x7f8b800323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8b880096a0 tx=0x7f8b88006b40).stop 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 msgr2=0x7f8b98106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8b8c005010 tx=0x7f8b8c00b040).stop 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 shutdown_connections 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8b8002ff10 0x7f8b800323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 --2- 172.21.15.112:0/2596156642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8b980f5410 0x7f8b98106810 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 >> 172.21.15.112:0/2596156642 conn(0x7f8b980f0b70 msgr2=0x7f8b9805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 shutdown_connections 2024-02-13T03:36:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:37.976+0000 7f8b9d967700 1 -- 172.21.15.112:0/2596156642 wait complete. 2024-02-13T03:36:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:36.814901+0000 mgr.smithi080.vyzlfg (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:38.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:37.976028+0000 mon.smithi080 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.112:0/2596156642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:39.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:39.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:38.815275+0000 mgr.smithi080.vyzlfg (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:41.573 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 -- 172.21.15.112:0/3499839375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 msgr2=0x7fd10c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 --2- 172.21.15.112:0/3499839375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c0f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd100009fb0 tx=0x7fd100009b30).stop 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 -- 172.21.15.112:0/3499839375 shutdown_connections 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 --2- 172.21.15.112:0/3499839375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c0f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 -- 172.21.15.112:0/3499839375 >> 172.21.15.112:0/3499839375 conn(0x7fd10c0f1c70 msgr2=0x7fd10c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 -- 172.21.15.112:0/3499839375 shutdown_connections 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.572+0000 7fd11319b700 1 -- 172.21.15.112:0/3499839375 wait complete. 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd11319b700 1 Processor -- start 2024-02-13T03:36:41.574 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd11319b700 1 -- start start 2024-02-13T03:36:41.576 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd11319b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd11319b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c106d00 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd110f37700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd110f37700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50760/0 (socket says 172.21.15.112:50760) 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.573+0000 7fd110f37700 1 -- 172.21.15.112:0/3340018752 learned_addr learned my addr 172.21.15.112:0/3340018752 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd110f37700 1 -- 172.21.15.112:0/3340018752 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100005040 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd110f37700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd100005010 tx=0x7fd10000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd100009320 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd100013380 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c106f00 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd100014e50 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.574+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c107390 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1000095f0 con 0x7fd10c0f6510 2024-02-13T03:36:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd1097fa700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 0x7fd0fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:41.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd100040e90 con 0x7fd10c0f6510 2024-02-13T03:36:41.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c045ad0 con 0x7fd10c0f6510 2024-02-13T03:36:41.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd10bfff700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 0x7fd0fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:41.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.575+0000 7fd10bfff700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 0x7fd0fc0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd0f80096a0 tx=0x7fd0f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:41.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.580+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd100014420 con 0x7fd10c0f6510 2024-02-13T03:36:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.783+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd10c108240 con 0x7fd10c0f6510 2024-02-13T03:36:41.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.783+0000 7fd1097fa700 1 -- 172.21.15.112:0/3340018752 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd100010230 con 0x7fd10c0f6510 2024-02-13T03:36:41.784 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:41.785 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:41.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 msgr2=0x7fd0fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:41.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 0x7fd0fc0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd0f80096a0 tx=0x7fd0f8006b40).stop 2024-02-13T03:36:41.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 msgr2=0x7fd10c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:41.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd100005010 tx=0x7fd10000b040).stop 2024-02-13T03:36:41.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 shutdown_connections 2024-02-13T03:36:41.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fd0fc02ff10 0x7fd0fc0323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:41.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 --2- 172.21.15.112:0/3340018752 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd10c0f6510 0x7fd10c106810 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:41.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 >> 172.21.15.112:0/3340018752 conn(0x7fd10c0f1c70 msgr2=0x7fd10c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:41.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 shutdown_connections 2024-02-13T03:36:41.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:41.785+0000 7fd11319b700 1 -- 172.21.15.112:0/3340018752 wait complete. 2024-02-13T03:36:41.787 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:40.815528+0000 mgr.smithi080.vyzlfg (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:41.784806+0000 mon.smithi080 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.112:0/3340018752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:43.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:43.560 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:42.815861+0000 mgr.smithi080.vyzlfg (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:45.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/570813814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 msgr2=0x7f4c680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/570813814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c680f90f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f4c60009960 tx=0x7f4c60004d10).stop 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/570813814 shutdown_connections 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/570813814 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c680f90f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/570813814 >> 172.21.15.112:0/570813814 conn(0x7f4c680f0b70 msgr2=0x7f4c680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/570813814 shutdown_connections 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.480+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/570813814 wait complete. 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6ebc0700 1 Processor -- start 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6ebc0700 1 -- start start 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6ebc0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6ebc0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68106c70 con 0x7f4c68060280 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6c95c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:45.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6c95c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50772/0 (socket says 172.21.15.112:50772) 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6c95c700 1 -- 172.21.15.112:0/707886040 learned_addr learned my addr 172.21.15.112:0/707886040 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.481+0000 7f4c6c95c700 1 -- 172.21.15.112:0/707886040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009640 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c6c95c700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f4c60007bd0 tx=0x7f4c60004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c60018070 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c6000b970 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c68106e70 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c6001d600 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c68107300 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.482+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c60024060 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.483+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c68045ad0 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.483+0000 7f4c657fa700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 0x7f4c50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.483+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c60041180 con 0x7f4c68060280 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.483+0000 7f4c67fff700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 0x7f4c50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.483+0000 7f4c67fff700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 0x7f4c50032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4c5c0096a0 tx=0x7f4c5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:45.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.487+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c600116a0 con 0x7f4c68060280 2024-02-13T03:36:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.691+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c681081b0 con 0x7f4c68060280 2024-02-13T03:36:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.692+0000 7f4c657fa700 1 -- 172.21.15.112:0/707886040 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c600118b0 con 0x7f4c68060280 2024-02-13T03:36:45.692 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:45.692 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 msgr2=0x7f4c50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 0x7f4c50032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4c5c0096a0 tx=0x7f4c5c006b40).stop 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 msgr2=0x7f4c68106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f4c60007bd0 tx=0x7f4c60004d10).stop 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 shutdown_connections 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f4c5002ff60 0x7f4c50032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 --2- 172.21.15.112:0/707886040 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c68060280 0x7f4c68106780 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 >> 172.21.15.112:0/707886040 conn(0x7f4c680f0b70 msgr2=0x7f4c680f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 shutdown_connections 2024-02-13T03:36:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:45.693+0000 7f4c6ebc0700 1 -- 172.21.15.112:0/707886040 wait complete. 2024-02-13T03:36:45.695 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:44.816134+0000 mgr.smithi080.vyzlfg (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:45.693045+0000 mon.smithi080 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.112:0/707886040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:47.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:47.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:46.816348+0000 mgr.smithi080.vyzlfg (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 -- 172.21.15.112:0/3167428079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 msgr2=0x7fe2cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3167428079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe2bc009fb0 tx=0x7fe2bc009b30).stop 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 -- 172.21.15.112:0/3167428079 shutdown_connections 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3167428079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 -- 172.21.15.112:0/3167428079 >> 172.21.15.112:0/3167428079 conn(0x7fe2cc0f1c70 msgr2=0x7fe2cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 -- 172.21.15.112:0/3167428079 shutdown_connections 2024-02-13T03:36:49.634 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.633+0000 7fe2d420a700 1 -- 172.21.15.112:0/3167428079 wait complete. 2024-02-13T03:36:49.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d420a700 1 Processor -- start 2024-02-13T03:36:49.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d420a700 1 -- start start 2024-02-13T03:36:49.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d420a700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d420a700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc106cd0 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d1fa6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d1fa6700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50788/0 (socket says 172.21.15.112:50788) 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.634+0000 7fe2d1fa6700 1 -- 172.21.15.112:0/3545881015 learned_addr learned my addr 172.21.15.112:0/3545881015 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2d1fa6700 1 -- 172.21.15.112:0/3545881015 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc005040 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2d1fa6700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe2bc005010 tx=0x7fe2bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2bc009320 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc106ed0 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2bc013380 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2bc014e50 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.635+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc107360 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2bc0263a0 con 0x7fe2cc0f6510 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2c27fc700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 0x7fe2b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:49.637 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2bc040e60 con 0x7fe2cc0f6510 2024-02-13T03:36:49.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc045ad0 con 0x7fe2cc0f6510 2024-02-13T03:36:49.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2d17a5700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 0x7fe2b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:49.638 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.636+0000 7fe2d17a5700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 0x7fe2b80323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe2c80096a0 tx=0x7fe2c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:49.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.640+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2bc012920 con 0x7fe2cc0f6510 2024-02-13T03:36:49.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.847+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2cc108080 con 0x7fe2cc0f6510 2024-02-13T03:36:49.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.848+0000 7fe2c27fc700 1 -- 172.21.15.112:0/3545881015 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2bc02b020 con 0x7fe2cc0f6510 2024-02-13T03:36:49.849 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:49.849 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:49.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 msgr2=0x7fe2b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:49.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 0x7fe2b80323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe2c80096a0 tx=0x7fe2c8006b40).stop 2024-02-13T03:36:49.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 msgr2=0x7fe2cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe2bc005010 tx=0x7fe2bc00b040).stop 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 shutdown_connections 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fe2b802ff10 0x7fe2b80323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 --2- 172.21.15.112:0/3545881015 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2cc0f6510 0x7fe2cc1067e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 >> 172.21.15.112:0/3545881015 conn(0x7fe2cc0f1c70 msgr2=0x7fe2cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 shutdown_connections 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:49.850+0000 7fe2d420a700 1 -- 172.21.15.112:0/3545881015 wait complete. 2024-02-13T03:36:49.851 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:48.816688+0000 mgr.smithi080.vyzlfg (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:49.849675+0000 mon.smithi080 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.112:0/3545881015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:51.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:51.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:50.816992+0000 mgr.smithi080.vyzlfg (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 -- 172.21.15.112:0/1771953262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 msgr2=0x7f0b080f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 --2- 172.21.15.112:0/1771953262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b080f9190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f0afc009fb0 tx=0x7f0afc009b30).stop 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 -- 172.21.15.112:0/1771953262 shutdown_connections 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 --2- 172.21.15.112:0/1771953262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b080f9190 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 -- 172.21.15.112:0/1771953262 >> 172.21.15.112:0/1771953262 conn(0x7f0b080f0b70 msgr2=0x7f0b080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 -- 172.21.15.112:0/1771953262 shutdown_connections 2024-02-13T03:36:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.534+0000 7f0b0d209700 1 -- 172.21.15.112:0/1771953262 wait complete. 2024-02-13T03:36:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b0d209700 1 Processor -- start 2024-02-13T03:36:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b0d209700 1 -- start start 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b0d209700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b0d209700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08106cc0 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b06d9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b06d9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:55894/0 (socket says 172.21.15.112:55894) 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.535+0000 7f0b06d9d700 1 -- 172.21.15.112:0/2536681046 learned_addr learned my addr 172.21.15.112:0/2536681046 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0b06d9d700 1 -- 172.21.15.112:0/2536681046 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc005040 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0b06d9d700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f0afc009830 tx=0x7f0afc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc018070 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b08106ec0 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0afc013410 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc01c3c0 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.536+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b08107350 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.537+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0afc01d420 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.537+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b08045ad0 con 0x7f0b080601e0 2024-02-13T03:36:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.537+0000 7f0aef7fe700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.537+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0afc041500 con 0x7f0b080601e0 2024-02-13T03:36:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.537+0000 7f0b0659c700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.538+0000 7f0b0659c700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 0x7f0af0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0af80096a0 tx=0x7f0af8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:53.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.541+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0afc013ca0 con 0x7f0b080601e0 2024-02-13T03:36:53.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.762+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b08108200 con 0x7f0b080601e0 2024-02-13T03:36:53.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.762+0000 7f0aef7fe700 1 -- 172.21.15.112:0/2536681046 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0afc01b370 con 0x7f0b080601e0 2024-02-13T03:36:53.763 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:53.763 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 msgr2=0x7f0af0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 0x7f0af0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0af80096a0 tx=0x7f0af8006b40).stop 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 msgr2=0x7f0b081067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f0afc009830 tx=0x7f0afc009b30).stop 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 shutdown_connections 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 --2- 172.21.15.112:0/2536681046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b080601e0 0x7f0b081067d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:53.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 >> 172.21.15.112:0/2536681046 conn(0x7f0b080f0b70 msgr2=0x7f0b080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:53.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.764+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 shutdown_connections 2024-02-13T03:36:53.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:53.765+0000 7f0b0d209700 1 -- 172.21.15.112:0/2536681046 wait complete. 2024-02-13T03:36:53.766 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:36:53.899 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:53 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:53.112706+0000 mon.smithi080 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T03:36:53.900 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:53 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:53.336230+0000 mon.smithi080 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:36:53.900 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:53 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:53.342395+0000 mon.smithi080 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:52.817329+0000 mgr.smithi080.vyzlfg (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:36:53.112113+0000 mgr.smithi080.vyzlfg (mgr.14162) 307 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi080 ... 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:35:00Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:35:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:36:15Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-13T03:36:54.942 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('uid/gid not found') 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: uid/gid not found 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: yield (conn, connr) 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: code, '\n'.join(err))) 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi080 ... 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi080 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:35:00Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:35:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.943 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr time="2024-02-13T03:36:15Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: Traceback (most recent call last): 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: main() 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: r = ctx.func(ctx) 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: return func(ctx) 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: File "/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-13T03:36:54.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: raise RuntimeError('uid/gid not found') 2024-02-13T03:36:54.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: RuntimeError: uid/gid not found 2024-02-13T03:36:54.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:53.590912+0000 mon.smithi080 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:36:54.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:53.763951+0000 mon.smithi080 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.112:0/2536681046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:36:56.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:36:56.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:36:56.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:54.817589+0000 mgr.smithi080.vyzlfg (mgr.14162) 308 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:56.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:55.163818+0000 mon.smithi080 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:36:57.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:56.863856+0000 mon.smithi080 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:36:57.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:56.870735+0000 mon.smithi080 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:36:58.440 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:56.817910+0000 mgr.smithi080.vyzlfg (mgr.14162) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 -- 172.21.15.112:0/284104820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 msgr2=0x7fc1580f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 --2- 172.21.15.112:0/284104820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1580f4b60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fc148009960 tx=0x7fc148004d10).stop 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 -- 172.21.15.112:0/284104820 shutdown_connections 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 --2- 172.21.15.112:0/284104820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1580f4b60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 -- 172.21.15.112:0/284104820 >> 172.21.15.112:0/284104820 conn(0x7fc1580f0260 msgr2=0x7fc1580f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 -- 172.21.15.112:0/284104820 shutdown_connections 2024-02-13T03:36:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.723+0000 7fc1600a5700 1 -- 172.21.15.112:0/284104820 wait complete. 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc1600a5700 1 Processor -- start 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc1600a5700 1 -- start start 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc1600a5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc1600a5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158106ca0 con 0x7fc1580f4740 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc15de41700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc15de41700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:55908/0 (socket says 172.21.15.112:55908) 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.724+0000 7fc15de41700 1 -- 172.21.15.112:0/2738546974 learned_addr learned my addr 172.21.15.112:0/2738546974 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc15de41700 1 -- 172.21.15.112:0/2738546974 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009640 con 0x7fc1580f4740 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc15de41700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fc148004b70 tx=0x7fc148004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc148018070 con 0x7fc1580f4740 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc158106ea0 con 0x7fc1580f4740 2024-02-13T03:36:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc14800b970 con 0x7fc1580f4740 2024-02-13T03:36:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14801c560 con 0x7fc1580f4740 2024-02-13T03:36:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.725+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc158107330 con 0x7fc1580f4740 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.726+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc14801c6c0 con 0x7fc1580f4740 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.726+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc158045ad0 con 0x7fc1580f4740 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.726+0000 7fc1567fc700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 0x7fc140032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.726+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc148041640 con 0x7fc1580f4740 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.726+0000 7fc15d640700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 0x7fc140032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:36:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.727+0000 7fc15d640700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 0x7fc140032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc14c0096a0 tx=0x7fc14c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:36:58.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.731+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1480127f0 con 0x7fc1580f4740 2024-02-13T03:36:58.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.947+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1581081e0 con 0x7fc1580f4740 2024-02-13T03:36:58.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.947+0000 7fc1567fc700 1 -- 172.21.15.112:0/2738546974 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1480127f0 con 0x7fc1580f4740 2024-02-13T03:36:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:36:58.948 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 msgr2=0x7fc140032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 0x7fc140032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc14c0096a0 tx=0x7fc14c006b40).stop 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 msgr2=0x7fc1581067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fc148004b70 tx=0x7fc148004d10).stop 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 shutdown_connections 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fc14002ffb0 0x7fc140032470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 --2- 172.21.15.112:0/2738546974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1580f4740 0x7fc1581067b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 >> 172.21.15.112:0/2738546974 conn(0x7fc1580f0260 msgr2=0x7fc1580f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:36:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 shutdown_connections 2024-02-13T03:36:58.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:36:58.950+0000 7fc1600a5700 1 -- 172.21.15.112:0/2738546974 wait complete. 2024-02-13T03:36:58.951 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:00.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:59 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:58.751378+0000 mon.smithi080 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:37:00.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:59 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:58.948876+0000 mon.smithi080 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.112:0/2738546974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:00.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:59 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:59.021820+0000 mon.smithi080 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:37:00.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:36:59 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:36:59.025592+0000 mon.smithi080 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' 2024-02-13T03:37:00.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:00.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:01.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:58.818281+0000 mgr.smithi080.vyzlfg (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:01.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:59.022667+0000 mgr.smithi080.vyzlfg (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:01.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cephadm 2024-02-13T03:36:59.027672+0000 mgr.smithi080.vyzlfg (mgr.14162) 312 : cephadm [INF] Deploying daemon alertmanager.smithi112 on smithi112 2024-02-13T03:37:01.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:36:59.749602+0000 mon.smithi080 (mon.0) 302 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- 172.21.15.112:0/101555452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 msgr2=0x7f06300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 --2- 172.21.15.112:0/101555452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f90f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f062c009fb0 tx=0x7f062c009b30).stop 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- 172.21.15.112:0/101555452 shutdown_connections 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 --2- 172.21.15.112:0/101555452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f90f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- 172.21.15.112:0/101555452 >> 172.21.15.112:0/101555452 conn(0x7f06300f0b70 msgr2=0x7f06300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- 172.21.15.112:0/101555452 shutdown_connections 2024-02-13T03:37:02.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- 172.21.15.112:0/101555452 wait complete. 2024-02-13T03:37:02.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 Processor -- start 2024-02-13T03:37:02.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.785+0000 7f0637be1700 1 -- start start 2024-02-13T03:37:02.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f0637be1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:02.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f0637be1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06300f4ff0 con 0x7f0630060280 2024-02-13T03:37:02.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f063597d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f063597d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43726/0 (socket says 172.21.15.112:43726) 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f063597d700 1 -- 172.21.15.112:0/491074398 learned_addr learned my addr 172.21.15.112:0/491074398 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f063597d700 1 -- 172.21.15.112:0/491074398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c005040 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f063597d700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f062c009830 tx=0x7f062c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.786+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f062c009320 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06300f5190 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f062c013380 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f062c014420 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06300f5620 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0630059100 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.787+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f062c01e420 con 0x7f0630060280 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.788+0000 7f06267fc700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:02.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.788+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f062c040ef0 con 0x7f0630060280 2024-02-13T03:37:02.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.788+0000 7f063517c700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:02.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.788+0000 7f063517c700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 0x7f061c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f06200096a0 tx=0x7f0620006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:02.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.792+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f062c01cb10 con 0x7f0630060280 2024-02-13T03:37:02.998 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.999+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0630045ad0 con 0x7f0630060280 2024-02-13T03:37:02.999 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:02.999+0000 7f06267fc700 1 -- 172.21.15.112:0/491074398 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f062c004490 con 0x7f0630060280 2024-02-13T03:37:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:02.999 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 msgr2=0x7f061c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 0x7f061c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f06200096a0 tx=0x7f0620006b40).stop 2024-02-13T03:37:03.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 msgr2=0x7f06300f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f062c009830 tx=0x7f062c00b040).stop 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 shutdown_connections 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f061c02ff60 0x7f061c032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 --2- 172.21.15.112:0/491074398 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0630060280 0x7f06300f6b70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 >> 172.21.15.112:0/491074398 conn(0x7f06300f0b70 msgr2=0x7f06300f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 shutdown_connections 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:03.001+0000 7f0637be1700 1 -- 172.21.15.112:0/491074398 wait complete. 2024-02-13T03:37:03.001 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:03.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:01.022939+0000 mgr.smithi080.vyzlfg (mgr.14162) 313 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:04.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:03 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:02.999627+0000 mon.smithi080 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.112:0/491074398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:04.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:04.806 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:05.131 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:03.023331+0000 mgr.smithi080.vyzlfg (mgr.14162) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:06.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 -- 172.21.15.112:0/3262650416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 msgr2=0x7f0fd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:06.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 --2- 172.21.15.112:0/3262650416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd40f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f0fc4009fb0 tx=0x7f0fc4009b30).stop 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 -- 172.21.15.112:0/3262650416 shutdown_connections 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 --2- 172.21.15.112:0/3262650416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd40f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 -- 172.21.15.112:0/3262650416 >> 172.21.15.112:0/3262650416 conn(0x7f0fd40f0b70 msgr2=0x7f0fd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 -- 172.21.15.112:0/3262650416 shutdown_connections 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.848+0000 7f0fdb524700 1 -- 172.21.15.112:0/3262650416 wait complete. 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fdb524700 1 Processor -- start 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fdb524700 1 -- start start 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fdb524700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fdb524700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4105bd0 con 0x7f0fd40f5410 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fd92c0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fd92c0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43754/0 (socket says 172.21.15.112:43754) 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fd92c0700 1 -- 172.21.15.112:0/457338184 learned_addr learned my addr 172.21.15.112:0/457338184 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.849+0000 7f0fd92c0700 1 -- 172.21.15.112:0/457338184 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4005040 con 0x7f0fd40f5410 2024-02-13T03:37:06.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fd92c0700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0fc4005010 tx=0x7f0fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:06.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc4009320 con 0x7f0fd40f5410 2024-02-13T03:37:06.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd4105dd0 con 0x7f0fd40f5410 2024-02-13T03:37:06.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fc4013380 con 0x7f0fd40f5410 2024-02-13T03:37:06.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc4014e50 con 0x7f0fd40f5410 2024-02-13T03:37:06.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.850+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd4106260 con 0x7f0fd40f5410 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.851+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fc40095f0 con 0x7f0fd40f5410 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.851+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd4045ad0 con 0x7f0fd40f5410 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.851+0000 7f0fc9ffb700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 0x7f0fc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.851+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fc4040ed0 con 0x7f0fd40f5410 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.851+0000 7f0fd8abf700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 0x7f0fc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:06.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.852+0000 7f0fd8abf700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 0x7f0fc00323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:06.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:06.856+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fc4014420 con 0x7f0fd40f5410 2024-02-13T03:37:07.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.061+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fd4107110 con 0x7f0fd40f5410 2024-02-13T03:37:07.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.062+0000 7f0fc9ffb700 1 -- 172.21.15.112:0/457338184 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fc4010230 con 0x7f0fd40f5410 2024-02-13T03:37:07.062 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:07.062 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:07.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 msgr2=0x7f0fc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:07.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 0x7f0fc00323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).stop 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 msgr2=0x7f0fd41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0fc4005010 tx=0x7f0fc400b040).stop 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 shutdown_connections 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0fc002ff10 0x7f0fc00323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 --2- 172.21.15.112:0/457338184 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fd40f5410 0x7f0fd41056e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 >> 172.21.15.112:0/457338184 conn(0x7f0fd40f0b70 msgr2=0x7f0fd40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 shutdown_connections 2024-02-13T03:37:07.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:07.064+0000 7f0fdb524700 1 -- 172.21.15.112:0/457338184 wait complete. 2024-02-13T03:37:07.065 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:07.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:05.023630+0000 mgr.smithi080.vyzlfg (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:08.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:07 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:07.062280+0000 mon.smithi080 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.112:0/457338184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:08.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:08.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:09.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:07.023947+0000 mgr.smithi080.vyzlfg (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:10.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 -- 172.21.15.112:0/1313535770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 msgr2=0x7f14040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:10.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 --2- 172.21.15.112:0/1313535770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14040f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f13f8009fb0 tx=0x7f13f8009b30).stop 2024-02-13T03:37:10.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 -- 172.21.15.112:0/1313535770 shutdown_connections 2024-02-13T03:37:10.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 --2- 172.21.15.112:0/1313535770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14040f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:10.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 -- 172.21.15.112:0/1313535770 >> 172.21.15.112:0/1313535770 conn(0x7f14040f0b70 msgr2=0x7f14040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:10.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 -- 172.21.15.112:0/1313535770 shutdown_connections 2024-02-13T03:37:10.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.699+0000 7f140a83c700 1 -- 172.21.15.112:0/1313535770 wait complete. 2024-02-13T03:37:10.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f140a83c700 1 Processor -- start 2024-02-13T03:37:10.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f140a83c700 1 -- start start 2024-02-13T03:37:10.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f140a83c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:10.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f140a83c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1404106cc0 con 0x7f14040f5410 2024-02-13T03:37:10.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f1403fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f1403fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43772/0 (socket says 172.21.15.112:43772) 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.700+0000 7f1403fff700 1 -- 172.21.15.112:0/4091177898 learned_addr learned my addr 172.21.15.112:0/4091177898 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f1403fff700 1 -- 172.21.15.112:0/4091177898 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8005040 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f1403fff700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f13f8005010 tx=0x7f13f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f8009320 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1404106ec0 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13f8013380 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f8014e50 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.701+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1404107350 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.702+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13f80263a0 con 0x7f14040f5410 2024-02-13T03:37:10.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.702+0000 7f1400ff9700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 0x7f13ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:10.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.702+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1404045ad0 con 0x7f14040f5410 2024-02-13T03:37:10.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.702+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13f8040fe0 con 0x7f14040f5410 2024-02-13T03:37:10.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.702+0000 7f14037fe700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 0x7f13ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:10.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.703+0000 7f14037fe700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 0x7f13ec0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f13f40096a0 tx=0x7f13f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:10.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.707+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13f8012920 con 0x7f14040f5410 2024-02-13T03:37:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.910+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1404108200 con 0x7f14040f5410 2024-02-13T03:37:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.911+0000 7f1400ff9700 1 -- 172.21.15.112:0/4091177898 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13f8012330 con 0x7f14040f5410 2024-02-13T03:37:10.912 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:10.912 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 msgr2=0x7f13ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 0x7f13ec0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f13f40096a0 tx=0x7f13f4006b40).stop 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 msgr2=0x7f14041067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f13f8005010 tx=0x7f13f800b040).stop 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 shutdown_connections 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f13ec02ff10 0x7f13ec0323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 --2- 172.21.15.112:0/4091177898 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f14040f5410 0x7f14041067d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 >> 172.21.15.112:0/4091177898 conn(0x7f14040f0b70 msgr2=0x7f140405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:10.914 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 shutdown_connections 2024-02-13T03:37:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:10.913+0000 7f140a83c700 1 -- 172.21.15.112:0/4091177898 wait complete. 2024-02-13T03:37:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:11.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:09.024257+0000 mgr.smithi080.vyzlfg (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:12.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:11 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:10.912672+0000 mon.smithi080 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.112:0/4091177898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:12.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:12.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:13.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:11.024528+0000 mgr.smithi080.vyzlfg (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 -- 172.21.15.112:0/3274838192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 msgr2=0x7f8aa00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/3274838192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa00f4be0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8a90009fb0 tx=0x7f8a90009b30).stop 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 -- 172.21.15.112:0/3274838192 shutdown_connections 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/3274838192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa00f4be0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 -- 172.21.15.112:0/3274838192 >> 172.21.15.112:0/3274838192 conn(0x7f8aa00f02e0 msgr2=0x7f8aa00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 -- 172.21.15.112:0/3274838192 shutdown_connections 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.709+0000 7f8aa56b7700 1 -- 172.21.15.112:0/3274838192 wait complete. 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8aa56b7700 1 Processor -- start 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8aa56b7700 1 -- start start 2024-02-13T03:37:14.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8aa56b7700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8aa56b7700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa0106cc0 con 0x7f8aa00f47c0 2024-02-13T03:37:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8a9effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8a9effd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45652/0 (socket says 172.21.15.112:45652) 2024-02-13T03:37:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.710+0000 7f8a9effd700 1 -- 172.21.15.112:0/2438459575 learned_addr learned my addr 172.21.15.112:0/2438459575 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8a9effd700 1 -- 172.21.15.112:0/2438459575 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90005040 con 0x7f8aa00f47c0 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8a9effd700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8a90005010 tx=0x7f8a9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a90009320 con 0x7f8aa00f47c0 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa0106ec0 con 0x7f8aa00f47c0 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a90013380 con 0x7f8aa00f47c0 2024-02-13T03:37:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a90014e50 con 0x7f8aa00f47c0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.711+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0107350 con 0x7f8aa00f47c0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.712+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a900095f0 con 0x7f8aa00f47c0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.712+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa0045ad0 con 0x7f8aa00f47c0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.712+0000 7f8a877fe700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 0x7f8a880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.712+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a90040fa0 con 0x7f8aa00f47c0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.712+0000 7f8a9e7fc700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 0x7f8a880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:14.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.713+0000 7f8a9e7fc700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 0x7f8a880323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f8a940096a0 tx=0x7f8a94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:14.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.717+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a90014420 con 0x7f8aa00f47c0 2024-02-13T03:37:14.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.921+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8aa0108200 con 0x7f8aa00f47c0 2024-02-13T03:37:14.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.921+0000 7f8a877fe700 1 -- 172.21.15.112:0/2438459575 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a90010230 con 0x7f8aa00f47c0 2024-02-13T03:37:14.922 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:14.922 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 msgr2=0x7f8a880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 0x7f8a880323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f8a940096a0 tx=0x7f8a94006b40).stop 2024-02-13T03:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 msgr2=0x7f8aa01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8a90005010 tx=0x7f8a9000b040).stop 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 shutdown_connections 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f8a8802ff10 0x7f8a880323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 --2- 172.21.15.112:0/2438459575 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8aa00f47c0 0x7f8aa01067d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 >> 172.21.15.112:0/2438459575 conn(0x7f8aa00f02e0 msgr2=0x7f8aa00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 shutdown_connections 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:14.923+0000 7f8aa56b7700 1 -- 172.21.15.112:0/2438459575 wait complete. 2024-02-13T03:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:15.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:13.024844+0000 mgr.smithi080.vyzlfg (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:16.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:15 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:14.922740+0000 mon.smithi080 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.112:0/2438459575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:16.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:16.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:17.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:15.025237+0000 mgr.smithi080.vyzlfg (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:18.569 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 -- 172.21.15.112:0/2983631094 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 msgr2=0x7f1ab80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 --2- 172.21.15.112:0/2983631094 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab80f90f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f1aa8009fb0 tx=0x7f1aa8009b30).stop 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 -- 172.21.15.112:0/2983631094 shutdown_connections 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 --2- 172.21.15.112:0/2983631094 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab80f90f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 -- 172.21.15.112:0/2983631094 >> 172.21.15.112:0/2983631094 conn(0x7f1ab80f0b70 msgr2=0x7f1ab80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 -- 172.21.15.112:0/2983631094 shutdown_connections 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.543+0000 7f1abe708700 1 -- 172.21.15.112:0/2983631094 wait complete. 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1abe708700 1 Processor -- start 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1abe708700 1 -- start start 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1abe708700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1abe708700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8106c50 con 0x7f1ab8060280 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1ab7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:18.570 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1ab7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45682/0 (socket says 172.21.15.112:45682) 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.544+0000 7f1ab7fff700 1 -- 172.21.15.112:0/103394675 learned_addr learned my addr 172.21.15.112:0/103394675 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1ab7fff700 1 -- 172.21.15.112:0/103394675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa8005040 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1ab7fff700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1aa80089f0 tx=0x7f1aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa8009320 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab8106e50 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aa8013380 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa8014e20 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.545+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab81072e0 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1aa80095f0 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab8045ad0 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1ab4ff9700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 0x7f1aa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1aa8040f50 con 0x7f1ab8060280 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1ab77fe700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 0x7f1aa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:18.571 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.546+0000 7f1ab77fe700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 0x7f1aa00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1aac0096a0 tx=0x7f1aac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:18.572 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.551+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1aa8014420 con 0x7f1ab8060280 2024-02-13T03:37:18.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.755+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ab8108190 con 0x7f1ab8060280 2024-02-13T03:37:18.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.756+0000 7f1ab4ff9700 1 -- 172.21.15.112:0/103394675 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1aa801d980 con 0x7f1ab8060280 2024-02-13T03:37:18.756 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:18.757 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 msgr2=0x7f1aa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 0x7f1aa00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1aac0096a0 tx=0x7f1aac006b40).stop 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 msgr2=0x7f1ab8106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1aa80089f0 tx=0x7f1aa800b040).stop 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 shutdown_connections 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f1aa002ff10 0x7f1aa00323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:18.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 --2- 172.21.15.112:0/103394675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1ab8060280 0x7f1ab8106760 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:18.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 >> 172.21.15.112:0/103394675 conn(0x7f1ab80f0b70 msgr2=0x7f1ab80f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:18.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 shutdown_connections 2024-02-13T03:37:18.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:18.757+0000 7f1abe708700 1 -- 172.21.15.112:0/103394675 wait complete. 2024-02-13T03:37:18.759 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:19.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:17.025647+0000 mgr.smithi080.vyzlfg (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:19.191 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:18.757185+0000 mon.smithi080 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.112:0/103394675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:20.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:20.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:21.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:19.025997+0000 mgr.smithi080.vyzlfg (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- 172.21.15.112:0/4153949021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 msgr2=0x7f0d700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 --2- 172.21.15.112:0/4153949021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d700f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f0d6c009fb0 tx=0x7f0d6c009b30).stop 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- 172.21.15.112:0/4153949021 shutdown_connections 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 --2- 172.21.15.112:0/4153949021 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d700f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- 172.21.15.112:0/4153949021 >> 172.21.15.112:0/4153949021 conn(0x7f0d700f0b70 msgr2=0x7f0d700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- 172.21.15.112:0/4153949021 shutdown_connections 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- 172.21.15.112:0/4153949021 wait complete. 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 Processor -- start 2024-02-13T03:37:22.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 -- start start 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.271+0000 7f0d77464700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d77464700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d70106cd0 con 0x7f0d700f5410 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d75200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d75200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50590/0 (socket says 172.21.15.112:50590) 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d75200700 1 -- 172.21.15.112:0/177788675 learned_addr learned my addr 172.21.15.112:0/177788675 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d75200700 1 -- 172.21.15.112:0/177788675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c005040 con 0x7f0d700f5410 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d75200700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f0d6c005010 tx=0x7f0d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.272+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d6c009320 con 0x7f0d700f5410 2024-02-13T03:37:22.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.273+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d70106ed0 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.273+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d6c013380 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.273+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d6c014e50 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.273+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d70107360 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d6c0095f0 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d70045ad0 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d65ffb700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d6c040e70 con 0x7f0d700f5410 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d749ff700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.274+0000 7f0d749ff700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0d600096a0 tx=0x7f0d60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:22.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.279+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d6c01dbc0 con 0x7f0d700f5410 2024-02-13T03:37:22.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.484+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d700fafa0 con 0x7f0d700f5410 2024-02-13T03:37:22.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.485+0000 7f0d65ffb700 1 -- 172.21.15.112:0/177788675 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d6c010230 con 0x7f0d700f5410 2024-02-13T03:37:22.486 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:22.486 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:22.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 msgr2=0x7f0d5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:22.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0d600096a0 tx=0x7f0d60006b40).stop 2024-02-13T03:37:22.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 msgr2=0x7f0d701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f0d6c005010 tx=0x7f0d6c00b040).stop 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 shutdown_connections 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 --2- 172.21.15.112:0/177788675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0d700f5410 0x7f0d701067e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 >> 172.21.15.112:0/177788675 conn(0x7f0d700f0b70 msgr2=0x7f0d7005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 shutdown_connections 2024-02-13T03:37:22.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:22.487+0000 7f0d77464700 1 -- 172.21.15.112:0/177788675 wait complete. 2024-02-13T03:37:22.489 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:23.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:21.026367+0000 mgr.smithi080.vyzlfg (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:23.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:22.486604+0000 mon.smithi080 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.112:0/177788675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:24.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:24.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:25.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:23.026764+0000 mgr.smithi080.vyzlfg (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 -- 172.21.15.112:0/2418175199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 msgr2=0x7f42900f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 --2- 172.21.15.112:0/2418175199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42900f4b60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4280009fb0 tx=0x7f4280009b30).stop 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 -- 172.21.15.112:0/2418175199 shutdown_connections 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 --2- 172.21.15.112:0/2418175199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42900f4b60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 -- 172.21.15.112:0/2418175199 >> 172.21.15.112:0/2418175199 conn(0x7f42900f0260 msgr2=0x7f42900f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 -- 172.21.15.112:0/2418175199 shutdown_connections 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.152+0000 7f4297e99700 1 -- 172.21.15.112:0/2418175199 wait complete. 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4297e99700 1 Processor -- start 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4297e99700 1 -- start start 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4297e99700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4297e99700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290106ca0 con 0x7f42900f4740 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4295c35700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4295c35700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50612/0 (socket says 172.21.15.112:50612) 2024-02-13T03:37:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4295c35700 1 -- 172.21.15.112:0/2223096049 learned_addr learned my addr 172.21.15.112:0/2223096049 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.153+0000 7f4295c35700 1 -- 172.21.15.112:0/2223096049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280005040 con 0x7f42900f4740 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f4295c35700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f428000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280009320 con 0x7f42900f4740 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4290106ea0 con 0x7f42900f4740 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4280013380 con 0x7f42900f4740 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280014e50 con 0x7f42900f4740 2024-02-13T03:37:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.154+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4290107330 con 0x7f42900f4740 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42800095f0 con 0x7f42900f4740 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4290045ad0 con 0x7f42900f4740 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f42867fc700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 0x7f427c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4280040e70 con 0x7f42900f4740 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f4295434700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 0x7f427c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.155+0000 7f4295434700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 0x7f427c0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f428c0096a0 tx=0x7f428c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:26.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.160+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4280014420 con 0x7f42900f4740 2024-02-13T03:37:26.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.366+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42901081e0 con 0x7f42900f4740 2024-02-13T03:37:26.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.366+0000 7f42867fc700 1 -- 172.21.15.112:0/2223096049 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4280010230 con 0x7f42900f4740 2024-02-13T03:37:26.367 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:26.367 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:26.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 msgr2=0x7f427c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 0x7f427c0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f428c0096a0 tx=0x7f428c006b40).stop 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 msgr2=0x7f42901067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f428000b040).stop 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 shutdown_connections 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f427c02ff10 0x7f427c0323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 --2- 172.21.15.112:0/2223096049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f42900f4740 0x7f42901067b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 >> 172.21.15.112:0/2223096049 conn(0x7f42900f0260 msgr2=0x7f42900f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 shutdown_connections 2024-02-13T03:37:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:26.368+0000 7f4297e99700 1 -- 172.21.15.112:0/2223096049 wait complete. 2024-02-13T03:37:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:27.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:25.027031+0000 mgr.smithi080.vyzlfg (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:27.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:26.367783+0000 mon.smithi080 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.112:0/2223096049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:27.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:27.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:29.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:27.027418+0000 mgr.smithi080.vyzlfg (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:29.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 -- 172.21.15.112:0/3189363142 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 msgr2=0x7fee180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 --2- 172.21.15.112:0/3189363142 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee180f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fee0c009fb0 tx=0x7fee0c009b30).stop 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 -- 172.21.15.112:0/3189363142 shutdown_connections 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 --2- 172.21.15.112:0/3189363142 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee180f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 -- 172.21.15.112:0/3189363142 >> 172.21.15.112:0/3189363142 conn(0x7fee180f0b70 msgr2=0x7fee180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 -- 172.21.15.112:0/3189363142 shutdown_connections 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.891+0000 7fee1edf1700 1 -- 172.21.15.112:0/3189363142 wait complete. 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1edf1700 1 Processor -- start 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1edf1700 1 -- start start 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1edf1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1edf1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18105bd0 con 0x7fee180f5410 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1cb8d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:29.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1cb8d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:50626/0 (socket says 172.21.15.112:50626) 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.892+0000 7fee1cb8d700 1 -- 172.21.15.112:0/2018341719 learned_addr learned my addr 172.21.15.112:0/2018341719 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee1cb8d700 1 -- 172.21.15.112:0/2018341719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c005040 con 0x7fee180f5410 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee1cb8d700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fee0c005010 tx=0x7fee0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee0c009320 con 0x7fee180f5410 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee18105dd0 con 0x7fee180f5410 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee0c013380 con 0x7fee180f5410 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee0c014e50 con 0x7fee180f5410 2024-02-13T03:37:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.893+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee18106260 con 0x7fee180f5410 2024-02-13T03:37:29.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.894+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee0c014420 con 0x7fee180f5410 2024-02-13T03:37:29.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.894+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18045ad0 con 0x7fee180f5410 2024-02-13T03:37:29.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.894+0000 7fee157fa700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 0x7fee080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:29.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.894+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee0c040e30 con 0x7fee180f5410 2024-02-13T03:37:29.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.894+0000 7fee17fff700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 0x7fee080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:29.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.895+0000 7fee17fff700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 0x7fee080323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fee040096a0 tx=0x7fee04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:29.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:29.899+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee0c0095f0 con 0x7fee180f5410 2024-02-13T03:37:30.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.103+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee18107110 con 0x7fee180f5410 2024-02-13T03:37:30.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.104+0000 7fee157fa700 1 -- 172.21.15.112:0/2018341719 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee0c01db70 con 0x7fee180f5410 2024-02-13T03:37:30.104 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:30.104 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 msgr2=0x7fee080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 0x7fee080323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fee040096a0 tx=0x7fee04006b40).stop 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 msgr2=0x7fee181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fee0c005010 tx=0x7fee0c00b040).stop 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 shutdown_connections 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fee0802ff10 0x7fee080323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 --2- 172.21.15.112:0/2018341719 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee180f5410 0x7fee181056e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 >> 172.21.15.112:0/2018341719 conn(0x7fee180f0b70 msgr2=0x7fee180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 shutdown_connections 2024-02-13T03:37:30.106 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:30.105+0000 7fee1edf1700 1 -- 172.21.15.112:0/2018341719 wait complete. 2024-02-13T03:37:30.107 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:31.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:29.027826+0000 mgr.smithi080.vyzlfg (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:31.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:30.105074+0000 mon.smithi080 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.112:0/2018341719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:31.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:31.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:33.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:31.028220+0000 mgr.smithi080.vyzlfg (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 -- 172.21.15.112:0/1298175829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 msgr2=0x7f89e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 --2- 172.21.15.112:0/1298175829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e80f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f89dc009fb0 tx=0x7f89dc009b30).stop 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 -- 172.21.15.112:0/1298175829 shutdown_connections 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 --2- 172.21.15.112:0/1298175829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e80f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 -- 172.21.15.112:0/1298175829 >> 172.21.15.112:0/1298175829 conn(0x7f89e80f0b70 msgr2=0x7f89e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 -- 172.21.15.112:0/1298175829 shutdown_connections 2024-02-13T03:37:33.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.700+0000 7f89eec2e700 1 -- 172.21.15.112:0/1298175829 wait complete. 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89eec2e700 1 Processor -- start 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89eec2e700 1 -- start start 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89eec2e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89eec2e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8106cc0 con 0x7f89e80f5410 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89ec9ca700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89ec9ca700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47804/0 (socket says 172.21.15.112:47804) 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.701+0000 7f89ec9ca700 1 -- 172.21.15.112:0/2584001638 learned_addr learned my addr 172.21.15.112:0/2584001638 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:33.703 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89ec9ca700 1 -- 172.21.15.112:0/2584001638 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc005040 con 0x7f89e80f5410 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89ec9ca700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f89dc005010 tx=0x7f89dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89dc009320 con 0x7f89e80f5410 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89dc013380 con 0x7f89e80f5410 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8106ec0 con 0x7f89e80f5410 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89dc014e50 con 0x7f89e80f5410 2024-02-13T03:37:33.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.702+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e8107350 con 0x7f89e80f5410 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.703+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89dc014420 con 0x7f89e80f5410 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.703+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e8045ad0 con 0x7f89e80f5410 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.703+0000 7f89e57fa700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 0x7f89d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.703+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89dc040e30 con 0x7f89e80f5410 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.703+0000 7f89e7fff700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 0x7f89d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:33.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.704+0000 7f89e7fff700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 0x7f89d80323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f89d40096a0 tx=0x7f89d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:33.708 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.708+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89dc0095f0 con 0x7f89e80f5410 2024-02-13T03:37:33.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.912+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89e8108200 con 0x7f89e80f5410 2024-02-13T03:37:33.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.914+0000 7f89e57fa700 1 -- 172.21.15.112:0/2584001638 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89dc01db70 con 0x7f89e80f5410 2024-02-13T03:37:33.914 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:33.914 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:33.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.914+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 msgr2=0x7f89d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:33.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.914+0000 7f89eec2e700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 0x7f89d80323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f89d40096a0 tx=0x7f89d4006b40).stop 2024-02-13T03:37:33.916 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.914+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 msgr2=0x7f89e81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.914+0000 7f89eec2e700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f89dc005010 tx=0x7f89dc00b040).stop 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 shutdown_connections 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f89d802ff10 0x7f89d80323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 --2- 172.21.15.112:0/2584001638 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89e80f5410 0x7f89e81067d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 >> 172.21.15.112:0/2584001638 conn(0x7f89e80f0b70 msgr2=0x7f89e805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:33.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 shutdown_connections 2024-02-13T03:37:33.918 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:33.915+0000 7f89eec2e700 1 -- 172.21.15.112:0/2584001638 wait complete. 2024-02-13T03:37:33.918 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:35.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:33.028494+0000 mgr.smithi080.vyzlfg (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:35.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:33.914130+0000 mon.smithi080 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.112:0/2584001638' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:35.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:35.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:37.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:35.028846+0000 mgr.smithi080.vyzlfg (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:37.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 -- 172.21.15.112:0/1177153096 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 msgr2=0x7fb0480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:37.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 --2- 172.21.15.112:0/1177153096 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb0480f90f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fb038009960 tx=0x7fb038004d10).stop 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 -- 172.21.15.112:0/1177153096 shutdown_connections 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 --2- 172.21.15.112:0/1177153096 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb0480f90f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 -- 172.21.15.112:0/1177153096 >> 172.21.15.112:0/1177153096 conn(0x7fb0480f0b70 msgr2=0x7fb0480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 -- 172.21.15.112:0/1177153096 shutdown_connections 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.504+0000 7fb04df70700 1 -- 172.21.15.112:0/1177153096 wait complete. 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb04df70700 1 Processor -- start 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb04df70700 1 -- start start 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb04df70700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb04df70700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb048106c70 con 0x7fb048060280 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb0477fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb0477fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47826/0 (socket says 172.21.15.112:47826) 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb0477fe700 1 -- 172.21.15.112:0/1760738293 learned_addr learned my addr 172.21.15.112:0/1760738293 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.505+0000 7fb0477fe700 1 -- 172.21.15.112:0/1760738293 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038009640 con 0x7fb048060280 2024-02-13T03:37:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb0477fe700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fb038007bd0 tx=0x7fb038004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:37.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb038018070 con 0x7fb048060280 2024-02-13T03:37:37.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb048106e70 con 0x7fb048060280 2024-02-13T03:37:37.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb03800b970 con 0x7fb048060280 2024-02-13T03:37:37.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb03801d600 con 0x7fb048060280 2024-02-13T03:37:37.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.506+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb048107300 con 0x7fb048060280 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb038024060 con 0x7fb048060280 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb048045ad0 con 0x7fb048060280 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb03ffff700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb038041150 con 0x7fb048060280 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb046ffd700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:37.509 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.507+0000 7fb046ffd700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 0x7fb034032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb0300096a0 tx=0x7fb030006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:37.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.512+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb03801b970 con 0x7fb048060280 2024-02-13T03:37:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.714+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb048108150 con 0x7fb048060280 2024-02-13T03:37:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.715+0000 7fb03ffff700 1 -- 172.21.15.112:0/1760738293 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb03801b970 con 0x7fb048060280 2024-02-13T03:37:37.715 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:37.716 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 msgr2=0x7fb034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 0x7fb034032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb0300096a0 tx=0x7fb030006b40).stop 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 msgr2=0x7fb048106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fb038007bd0 tx=0x7fb038004d10).stop 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 shutdown_connections 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb03402ff60 0x7fb034032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 --2- 172.21.15.112:0/1760738293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb048060280 0x7fb048106780 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 >> 172.21.15.112:0/1760738293 conn(0x7fb0480f0b70 msgr2=0x7fb0480f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 shutdown_connections 2024-02-13T03:37:37.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:37.716+0000 7fb04df70700 1 -- 172.21.15.112:0/1760738293 wait complete. 2024-02-13T03:37:37.718 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:38.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:37 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:37.716230+0000 mon.smithi080 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.112:0/1760738293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:39.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:37.029255+0000 mgr.smithi080.vyzlfg (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:39.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:39.473 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:41.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:39.029660+0000 mgr.smithi080.vyzlfg (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:41.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- 172.21.15.112:0/3865015983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 msgr2=0x7f83840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:41.204 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 --2- 172.21.15.112:0/3865015983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f83840f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f8374009fb0 tx=0x7f8374009b30).stop 2024-02-13T03:37:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- 172.21.15.112:0/3865015983 shutdown_connections 2024-02-13T03:37:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 --2- 172.21.15.112:0/3865015983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f83840f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- 172.21.15.112:0/3865015983 >> 172.21.15.112:0/3865015983 conn(0x7f83840f0b70 msgr2=0x7f83840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- 172.21.15.112:0/3865015983 shutdown_connections 2024-02-13T03:37:41.205 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- 172.21.15.112:0/3865015983 wait complete. 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 Processor -- start 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.205+0000 7f838c17c700 1 -- start start 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f838c17c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f838c17c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8384060220 con 0x7f83840f5410 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f8389f18700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f8389f18700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47846/0 (socket says 172.21.15.112:47846) 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f8389f18700 1 -- 172.21.15.112:0/3338589107 learned_addr learned my addr 172.21.15.112:0/3338589107 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f8389f18700 1 -- 172.21.15.112:0/3338589107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374005040 con 0x7f83840f5410 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f8389f18700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8374005010 tx=0x7f837400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8374009320 con 0x7f83840f5410 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8384060420 con 0x7f83840f5410 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8374013380 con 0x7f83840f5410 2024-02-13T03:37:41.207 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8384109bb0 con 0x7f83840f5410 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.206+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8374014e50 con 0x7f83840f5410 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.207+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8374014420 con 0x7f83840f5410 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.207+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8384059100 con 0x7f83840f5410 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.207+0000 7f837a7fc700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 0x7f83700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.207+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8374040f70 con 0x7f83840f5410 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.207+0000 7f8389717700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 0x7f83700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:41.208 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.208+0000 7f8389717700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 0x7f83700323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f83800096a0 tx=0x7f8380006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:41.211 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.212+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f837401c320 con 0x7f83840f5410 2024-02-13T03:37:41.417 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.418+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83840fafa0 con 0x7f83840f5410 2024-02-13T03:37:41.418 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.418+0000 7f837a7fc700 1 -- 172.21.15.112:0/3338589107 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8374010230 con 0x7f83840f5410 2024-02-13T03:37:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:41.418 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 msgr2=0x7f83700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 0x7f83700323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f83800096a0 tx=0x7f8380006b40).stop 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 msgr2=0x7f8384109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8374005010 tx=0x7f837400b040).stop 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 shutdown_connections 2024-02-13T03:37:41.420 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f837002ff10 0x7f83700323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:41.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 --2- 172.21.15.112:0/3338589107 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83840f5410 0x7f8384109500 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:41.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 >> 172.21.15.112:0/3338589107 conn(0x7f83840f0b70 msgr2=0x7f83840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:41.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 shutdown_connections 2024-02-13T03:37:41.421 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:41.420+0000 7f838c17c700 1 -- 172.21.15.112:0/3338589107 wait complete. 2024-02-13T03:37:41.421 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:42.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:41 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:41.418824+0000 mon.smithi080 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.112:0/3338589107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:43.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:43.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:43.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:41.030000+0000 mgr.smithi080.vyzlfg (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:45.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 -- 172.21.15.112:0/2218886192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 msgr2=0x7fbe980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:45.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/2218886192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fbe88009fb0 tx=0x7fbe88009b30).stop 2024-02-13T03:37:45.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 -- 172.21.15.112:0/2218886192 shutdown_connections 2024-02-13T03:37:45.102 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/2218886192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 -- 172.21.15.112:0/2218886192 >> 172.21.15.112:0/2218886192 conn(0x7fbe980f0b70 msgr2=0x7fbe980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 -- 172.21.15.112:0/2218886192 shutdown_connections 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.101+0000 7fbe9fa08700 1 -- 172.21.15.112:0/2218886192 wait complete. 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9fa08700 1 Processor -- start 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9fa08700 1 -- start start 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9fa08700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9fa08700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980fb960 con 0x7fbe980f5410 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9d7a4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9d7a4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:36790/0 (socket says 172.21.15.112:36790) 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9d7a4700 1 -- 172.21.15.112:0/3240681062 learned_addr learned my addr 172.21.15.112:0/3240681062 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9d7a4700 1 -- 172.21.15.112:0/3240681062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe88005040 con 0x7fbe980f5410 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.102+0000 7fbe9d7a4700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbe88005010 tx=0x7fbe8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:45.103 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.103+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe88009320 con 0x7fbe980f5410 2024-02-13T03:37:45.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.103+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe980fbb00 con 0x7fbe980f5410 2024-02-13T03:37:45.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.103+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe88013380 con 0x7fbe980f5410 2024-02-13T03:37:45.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.103+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe88014e50 con 0x7fbe980f5410 2024-02-13T03:37:45.104 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.103+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe980fbf90 con 0x7fbe980f5410 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.104+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe88014420 con 0x7fbe980f5410 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.104+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe98059100 con 0x7fbe980f5410 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.104+0000 7fbe8dffb700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.104+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe88040e60 con 0x7fbe980f5410 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.104+0000 7fbe9cfa3700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:45.105 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.105+0000 7fbe9cfa3700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 0x7fbe840323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbe940096a0 tx=0x7fbe94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:45.108 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.109+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe880095f0 con 0x7fbe980f5410 2024-02-13T03:37:45.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:43.030346+0000 mgr.smithi080.vyzlfg (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:45.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.314+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe98045ad0 con 0x7fbe980f5410 2024-02-13T03:37:45.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.315+0000 7fbe8dffb700 1 -- 172.21.15.112:0/3240681062 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe8801db70 con 0x7fbe980f5410 2024-02-13T03:37:45.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:45.315 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:45.316 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 msgr2=0x7fbe840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 0x7fbe840323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbe940096a0 tx=0x7fbe94006b40).stop 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 msgr2=0x7fbe980fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbe88005010 tx=0x7fbe8800b040).stop 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 shutdown_connections 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe8402ff10 0x7fbe840323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 --2- 172.21.15.112:0/3240681062 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe980f5410 0x7fbe980fd4e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 >> 172.21.15.112:0/3240681062 conn(0x7fbe980f0b70 msgr2=0x7fbe9805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 shutdown_connections 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:45.317+0000 7fbe9fa08700 1 -- 172.21.15.112:0/3240681062 wait complete. 2024-02-13T03:37:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:46.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:45 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:45.315357+0000 mon.smithi080 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.112:0/3240681062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:47.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:47.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:47.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:45.030735+0000 mgr.smithi080.vyzlfg (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:49.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 -- 172.21.15.112:0/184807579 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 msgr2=0x7ff2f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:49.091 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/184807579 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f90f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7ff2d8009fb0 tx=0x7ff2d8009b30).stop 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 -- 172.21.15.112:0/184807579 shutdown_connections 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/184807579 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f90f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 -- 172.21.15.112:0/184807579 >> 172.21.15.112:0/184807579 conn(0x7ff2f00f0b70 msgr2=0x7ff2f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 -- 172.21.15.112:0/184807579 shutdown_connections 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.072+0000 7ff2f49a0700 1 -- 172.21.15.112:0/184807579 wait complete. 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2f49a0700 1 Processor -- start 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2f49a0700 1 -- start start 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2f49a0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2f49a0700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00f4ff0 con 0x7ff2f0060280 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:36808/0 (socket says 172.21.15.112:36808) 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2ee59c700 1 -- 172.21.15.112:0/1542592989 learned_addr learned my addr 172.21.15.112:0/1542592989 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:49.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2ee59c700 1 -- 172.21.15.112:0/1542592989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8005040 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.073+0000 7ff2ee59c700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7ff2d80089f0 tx=0x7ff2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.074+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8009320 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.074+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f00f5190 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.074+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2d8013380 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.074+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8014e20 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.074+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f00f5620 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2d80095f0 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f0059100 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2e6ffd700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 0x7ff2dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2d8040e90 con 0x7ff2f0060280 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2edd9b700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 0x7ff2dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.075+0000 7ff2edd9b700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 0x7ff2dc0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff2e00096a0 tx=0x7ff2e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:49.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.079+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2d8014420 con 0x7ff2f0060280 2024-02-13T03:37:49.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:47.031142+0000 mgr.smithi080.vyzlfg (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:49.286 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.286+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f00fc090 con 0x7ff2f0060280 2024-02-13T03:37:49.287 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.286+0000 7ff2e6ffd700 1 -- 172.21.15.112:0/1542592989 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2d801d980 con 0x7ff2f0060280 2024-02-13T03:37:49.287 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:49.287 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:49.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 msgr2=0x7ff2dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 0x7ff2dc0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff2e00096a0 tx=0x7ff2e0006b40).stop 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 msgr2=0x7ff2f00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7ff2d80089f0 tx=0x7ff2d800b040).stop 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 shutdown_connections 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff2dc02ff10 0x7ff2dc0323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 --2- 172.21.15.112:0/1542592989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff2f0060280 0x7ff2f00f6b70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 >> 172.21.15.112:0/1542592989 conn(0x7ff2f00f0b70 msgr2=0x7ff2f00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 shutdown_connections 2024-02-13T03:37:49.289 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:49.288+0000 7ff2f49a0700 1 -- 172.21.15.112:0/1542592989 wait complete. 2024-02-13T03:37:49.290 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:50.194 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:49 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:49.287752+0000 mon.smithi080 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.112:0/1542592989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:51.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:51.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:51.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:49.031497+0000 mgr.smithi080.vyzlfg (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:52.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:51.031902+0000 mgr.smithi080.vyzlfg (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:53.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.080+0000 7f388b5e9700 1 -- 172.21.15.112:0/1100681747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 msgr2=0x7f38840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:53.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.080+0000 7f388b5e9700 1 --2- 172.21.15.112:0/1100681747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f3874009fb0 tx=0x7f3874009b30).stop 2024-02-13T03:37:53.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.080+0000 7f388b5e9700 1 -- 172.21.15.112:0/1100681747 shutdown_connections 2024-02-13T03:37:53.082 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.080+0000 7f388b5e9700 1 --2- 172.21.15.112:0/1100681747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 -- 172.21.15.112:0/1100681747 >> 172.21.15.112:0/1100681747 conn(0x7f38840f0b70 msgr2=0x7f38840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 -- 172.21.15.112:0/1100681747 shutdown_connections 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 -- 172.21.15.112:0/1100681747 wait complete. 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 Processor -- start 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 -- start start 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f388b5e9700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840fb960 con 0x7f38840f5410 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f3889385700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f3889385700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45320/0 (socket says 172.21.15.112:45320) 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.081+0000 7f3889385700 1 -- 172.21.15.112:0/3657767000 learned_addr learned my addr 172.21.15.112:0/3657767000 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.082+0000 7f3889385700 1 -- 172.21.15.112:0/3657767000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874005040 con 0x7f38840f5410 2024-02-13T03:37:53.083 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.082+0000 7f3889385700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3874005010 tx=0x7f387400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:53.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.084+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3874009320 con 0x7f38840f5410 2024-02-13T03:37:53.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.084+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840fbb00 con 0x7f38840f5410 2024-02-13T03:37:53.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.084+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3874013380 con 0x7f38840f5410 2024-02-13T03:37:53.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.084+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3874014e50 con 0x7f38840f5410 2024-02-13T03:37:53.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.084+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840fbf90 con 0x7f38840f5410 2024-02-13T03:37:53.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884059100 con 0x7f38840f5410 2024-02-13T03:37:53.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38740095f0 con 0x7f38840f5410 2024-02-13T03:37:53.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f3879ffb700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 0x7f38700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:53.085 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3874040e70 con 0x7f38840f5410 2024-02-13T03:37:53.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f3888b84700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 0x7f38700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:53.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.085+0000 7f3888b84700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 0x7f38700323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f38800096a0 tx=0x7f3880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:53.090 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.089+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3874014420 con 0x7f38840f5410 2024-02-13T03:37:53.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.294+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38840fafa0 con 0x7f38840f5410 2024-02-13T03:37:53.295 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.295+0000 7f3879ffb700 1 -- 172.21.15.112:0/3657767000 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3874010230 con 0x7f38840f5410 2024-02-13T03:37:53.295 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:53.295 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 msgr2=0x7f38700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 0x7f38700323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f38800096a0 tx=0x7f3880006b40).stop 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 msgr2=0x7f38840fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3874005010 tx=0x7f387400b040).stop 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 shutdown_connections 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f387002ff10 0x7f38700323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 --2- 172.21.15.112:0/3657767000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f38840f5410 0x7f38840fd4e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 >> 172.21.15.112:0/3657767000 conn(0x7f38840f0b70 msgr2=0x7f388405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 shutdown_connections 2024-02-13T03:37:53.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:53.296+0000 7f388b5e9700 1 -- 172.21.15.112:0/3657767000 wait complete. 2024-02-13T03:37:53.298 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:54.443 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:53.032257+0000 mgr.smithi080.vyzlfg (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:54.443 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:53.296215+0000 mon.smithi080 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.112:0/3657767000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:55.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:55.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2024451766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 msgr2=0x7fa2d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2024451766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d80f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fa2c8009fb0 tx=0x7fa2c8009b30).stop 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2024451766 shutdown_connections 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2024451766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d80f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2024451766 >> 172.21.15.112:0/2024451766 conn(0x7fa2d80f1c70 msgr2=0x7fa2d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2024451766 shutdown_connections 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.929+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2024451766 wait complete. 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2de8d2700 1 Processor -- start 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2de8d2700 1 -- start start 2024-02-13T03:37:56.931 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2de8d2700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:56.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2de8d2700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8069e20 con 0x7fa2d80f6510 2024-02-13T03:37:56.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:56.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45332/0 (socket says 172.21.15.112:45332) 2024-02-13T03:37:56.932 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.930+0000 7fa2d7fff700 1 -- 172.21.15.112:0/2922039763 learned_addr learned my addr 172.21.15.112:0/2922039763 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:37:56.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2d7fff700 1 -- 172.21.15.112:0/2922039763 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8005040 con 0x7fa2d80f6510 2024-02-13T03:37:56.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2d7fff700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fa2c8005010 tx=0x7fa2c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:56.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c8009320 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d8069fc0 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2c8013380 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c8014e50 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.931+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d806a450 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.932+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2c80095f0 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.932+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d8059100 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.932+0000 7fa2d4ff9700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 0x7fa2c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.932+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2c8040fa0 con 0x7fa2d80f6510 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.932+0000 7fa2d77fe700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 0x7fa2c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:37:56.934 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.933+0000 7fa2d77fe700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 0x7fa2c00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa2cc0096a0 tx=0x7fa2cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:37:56.937 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:56.937+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2c8014420 con 0x7fa2d80f6510 2024-02-13T03:37:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:55.032526+0000 mgr.smithi080.vyzlfg (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:37:57.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.141+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2d80fc0a0 con 0x7fa2d80f6510 2024-02-13T03:37:57.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.142+0000 7fa2d4ff9700 1 -- 172.21.15.112:0/2922039763 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2c8010230 con 0x7fa2d80f6510 2024-02-13T03:37:57.142 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:37:57.142 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 msgr2=0x7fa2c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 0x7fa2c00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa2cc0096a0 tx=0x7fa2cc006b40).stop 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 msgr2=0x7fa2d806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fa2c8005010 tx=0x7fa2c800b040).stop 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 shutdown_connections 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa2c002ff10 0x7fa2c00323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 --2- 172.21.15.112:0/2922039763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa2d80f6510 0x7fa2d806b9a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 >> 172.21.15.112:0/2922039763 conn(0x7fa2d80f1c70 msgr2=0x7fa2d80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 shutdown_connections 2024-02-13T03:37:57.144 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:37:57.143+0000 7fa2de8d2700 1 -- 172.21.15.112:0/2922039763 wait complete. 2024-02-13T03:37:57.145 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:37:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:56.865289+0000 mon.smithi080 (mon.0) 317 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:37:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:56.871783+0000 mon.smithi080 (mon.0) 318 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:37:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:37:57.143056+0000 mon.smithi080 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.112:0/2922039763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:37:58.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:37:58.734 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:37:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:37:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:57.032891+0000 mgr.smithi080.vyzlfg (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- 172.21.15.112:0/314439498 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 msgr2=0x7f9e3c0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/314439498 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c0f8fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f9e2c009fb0 tx=0x7f9e2c009b30).stop 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- 172.21.15.112:0/314439498 shutdown_connections 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/314439498 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c0f8fc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- 172.21.15.112:0/314439498 >> 172.21.15.112:0/314439498 conn(0x7f9e3c0f09a0 msgr2=0x7f9e3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:00.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- 172.21.15.112:0/314439498 shutdown_connections 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- 172.21.15.112:0/314439498 wait complete. 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 Processor -- start 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.745+0000 7f9e3b59e700 1 -- start start 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3b59e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3b59e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c108ce0 con 0x7f9e3c060140 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:45346/0 (socket says 172.21.15.112:45346) 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3a59c700 1 -- 172.21.15.112:0/1376711585 learned_addr learned my addr 172.21.15.112:0/1376711585 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3a59c700 1 -- 172.21.15.112:0/1376711585 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c005040 con 0x7f9e3c060140 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.746+0000 7f9e3a59c700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9e2c005010 tx=0x7f9e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c009320 con 0x7f9e3c060140 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3c108ee0 con 0x7f9e3c060140 2024-02-13T03:38:00.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e2c013380 con 0x7f9e3c060140 2024-02-13T03:38:00.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c014e50 con 0x7f9e3c060140 2024-02-13T03:38:00.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c109370 con 0x7f9e3c060140 2024-02-13T03:38:00.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.747+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3c045ad0 con 0x7f9e3c060140 2024-02-13T03:38:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.748+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e2c0095f0 con 0x7f9e3c060140 2024-02-13T03:38:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.748+0000 7f9e2affd700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 0x7f9e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.748+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e2c040e70 con 0x7f9e3c060140 2024-02-13T03:38:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.748+0000 7f9e39d9b700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 0x7f9e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:00.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.748+0000 7f9e39d9b700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 0x7f9e240323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9e300096a0 tx=0x7f9e30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:00.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.753+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e2c014420 con 0x7f9e3c060140 2024-02-13T03:38:00.768 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:37:59.033292+0000 mgr.smithi080.vyzlfg (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:00.958 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.957+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e3c10a1c0 con 0x7f9e3c060140 2024-02-13T03:38:00.959 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.958+0000 7f9e2affd700 1 -- 172.21.15.112:0/1376711585 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e2c010230 con 0x7f9e3c060140 2024-02-13T03:38:00.959 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:00.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 msgr2=0x7f9e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 0x7f9e240323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f9e300096a0 tx=0x7f9e30006b40).stop 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 msgr2=0x7f9e3c1087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9e2c005010 tx=0x7f9e2c00b040).stop 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 shutdown_connections 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f9e2402ff10 0x7f9e240323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 --2- 172.21.15.112:0/1376711585 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9e3c060140 0x7f9e3c1087f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 >> 172.21.15.112:0/1376711585 conn(0x7f9e3c0f09a0 msgr2=0x7f9e3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:00.961 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 shutdown_connections 2024-02-13T03:38:00.962 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:00.960+0000 7f9e3b59e700 1 -- 172.21.15.112:0/1376711585 wait complete. 2024-02-13T03:38:00.962 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:01.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:01 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:00.959748+0000 mon.smithi080 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.112:0/1376711585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:02.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:02.710 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:01.033642+0000 mgr.smithi080.vyzlfg (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.692+0000 7f0434f6f700 1 -- 172.21.15.112:0/998364365 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04300f6240 msgr2=0x7f04300f6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.692+0000 7f0434f6f700 1 --2- 172.21.15.112:0/998364365 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04300f6240 0x7f04300f6660 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f0420009fb0 tx=0x7f0420009b30).stop 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- 172.21.15.112:0/998364365 shutdown_connections 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 --2- 172.21.15.112:0/998364365 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04300f6240 0x7f04300f6660 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- 172.21.15.112:0/998364365 >> 172.21.15.112:0/998364365 conn(0x7f04300f19c0 msgr2=0x7f04300f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- 172.21.15.112:0/998364365 shutdown_connections 2024-02-13T03:38:04.694 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- 172.21.15.112:0/998364365 wait complete. 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 Processor -- start 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- start start 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.693+0000 7f0434f6f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043006bd80 con 0x7f043006b960 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f042f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f042f7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:60578/0 (socket says 172.21.15.112:60578) 2024-02-13T03:38:04.695 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f042f7fe700 1 -- 172.21.15.112:0/3996457548 learned_addr learned my addr 172.21.15.112:0/3996457548 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:04.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f042f7fe700 1 -- 172.21.15.112:0/3996457548 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420005040 con 0x7f043006b960 2024-02-13T03:38:04.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f042f7fe700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0420009f80 tx=0x7f0420009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0420013380 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0420013790 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043006a310 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0420014420 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.694+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043006a800 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.695+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0420025060 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.695+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0430045ad0 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.695+0000 7f0417fff700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 0x7f0418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.695+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0420040d70 con 0x7f043006b960 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.695+0000 7f042effd700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 0x7f0418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:04.697 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.696+0000 7f042effd700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 0x7f0418032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f04240096a0 tx=0x7f0424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:04.700 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.700+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f042000f280 con 0x7f043006b960 2024-02-13T03:38:04.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.906+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f043010c840 con 0x7f043006b960 2024-02-13T03:38:04.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.907+0000 7f0417fff700 1 -- 172.21.15.112:0/3996457548 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f042002d2d0 con 0x7f043006b960 2024-02-13T03:38:04.907 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:04.907 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:04.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 msgr2=0x7f0418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:04.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 0x7f0418032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f04240096a0 tx=0x7f0424006b40).stop 2024-02-13T03:38:04.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 msgr2=0x7f0430069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:04.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0420009f80 tx=0x7f0420009b30).stop 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 shutdown_connections 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f041802ff60 0x7f0418032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 --2- 172.21.15.112:0/3996457548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f043006b960 0x7f0430069e20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 >> 172.21.15.112:0/3996457548 conn(0x7f04300f19c0 msgr2=0x7f04300f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 shutdown_connections 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:04.908+0000 7f0434f6f700 1 -- 172.21.15.112:0/3996457548 wait complete. 2024-02-13T03:38:04.910 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:03.034002+0000 mgr.smithi080.vyzlfg (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:05.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:05 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:04.908203+0000 mon.smithi080 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.112:0/3996457548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:06.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:06.613 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:05.034250+0000 mgr.smithi080.vyzlfg (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 -- 172.21.15.112:0/4050518337 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 msgr2=0x7f72280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 --2- 172.21.15.112:0/4050518337 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f72280f90f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f721c009fb0 tx=0x7f721c009b30).stop 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 -- 172.21.15.112:0/4050518337 shutdown_connections 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 --2- 172.21.15.112:0/4050518337 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f72280f90f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 -- 172.21.15.112:0/4050518337 >> 172.21.15.112:0/4050518337 conn(0x7f72280f0b70 msgr2=0x7f72280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 -- 172.21.15.112:0/4050518337 shutdown_connections 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 -- 172.21.15.112:0/4050518337 wait complete. 2024-02-13T03:38:08.290 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.288+0000 7f722dffb700 1 Processor -- start 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f722dffb700 1 -- start start 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f722dffb700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f722dffb700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7228106c70 con 0x7f7228060280 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f72277fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f72277fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:60590/0 (socket says 172.21.15.112:60590) 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f72277fe700 1 -- 172.21.15.112:0/1662402817 learned_addr learned my addr 172.21.15.112:0/1662402817 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.289+0000 7f72277fe700 1 -- 172.21.15.112:0/1662402817 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c005040 con 0x7f7228060280 2024-02-13T03:38:08.291 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f72277fe700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f721c0089f0 tx=0x7f721c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:08.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c009320 con 0x7f7228060280 2024-02-13T03:38:08.292 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7228106e70 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f721c013380 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c014e20 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.290+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7228107300 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f721c0095f0 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7228045ad0 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f720ffff700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 0x7f72100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f721c040e90 con 0x7f7228060280 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f7226ffd700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 0x7f72100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:08.293 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.291+0000 7f7226ffd700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 0x7f72100323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f72180096a0 tx=0x7f7218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:08.296 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.296+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f721c014420 con 0x7f7228060280 2024-02-13T03:38:08.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.498+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72281081b0 con 0x7f7228060280 2024-02-13T03:38:08.499 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.499+0000 7f720ffff700 1 -- 172.21.15.112:0/1662402817 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f721c01d980 con 0x7f7228060280 2024-02-13T03:38:08.500 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:08.500 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 msgr2=0x7f72100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 0x7f72100323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f72180096a0 tx=0x7f7218006b40).stop 2024-02-13T03:38:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 msgr2=0x7f7228106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f721c0089f0 tx=0x7f721c00b040).stop 2024-02-13T03:38:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 shutdown_connections 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.500+0000 7f722dffb700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f721002ff10 0x7f72100323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.501+0000 7f722dffb700 1 --2- 172.21.15.112:0/1662402817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7228060280 0x7f7228106780 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.501+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 >> 172.21.15.112:0/1662402817 conn(0x7f72280f0b70 msgr2=0x7f72280f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.501+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 shutdown_connections 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:08.501+0000 7f722dffb700 1 -- 172.21.15.112:0/1662402817 wait complete. 2024-02-13T03:38:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:07.034487+0000 mgr.smithi080.vyzlfg (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:09.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:09 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:08.500464+0000 mon.smithi080 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.112:0/1662402817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:10.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:10.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:09.034742+0000 mgr.smithi080.vyzlfg (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- 172.21.15.112:0/4173459310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 msgr2=0x7fbe280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 --2- 172.21.15.112:0/4173459310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe280f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbe18009fb0 tx=0x7fbe18009b30).stop 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- 172.21.15.112:0/4173459310 shutdown_connections 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 --2- 172.21.15.112:0/4173459310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe280f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- 172.21.15.112:0/4173459310 >> 172.21.15.112:0/4173459310 conn(0x7fbe280f0b70 msgr2=0x7fbe280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- 172.21.15.112:0/4173459310 shutdown_connections 2024-02-13T03:38:12.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- 172.21.15.112:0/4173459310 wait complete. 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 Processor -- start 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- start start 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.271+0000 7fbe301e3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe28060220 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe2df7f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe2df7f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43976/0 (socket says 172.21.15.112:43976) 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe2df7f700 1 -- 172.21.15.112:0/3292408536 learned_addr learned my addr 172.21.15.112:0/3292408536 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe2df7f700 1 -- 172.21.15.112:0/3292408536 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18005040 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe2df7f700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbe180089f0 tx=0x7fbe1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe18009320 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe18013380 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe28060420 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe1801ed80 con 0x7fbe280f5410 2024-02-13T03:38:12.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.272+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe28109bb0 con 0x7fbe280f5410 2024-02-13T03:38:12.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.273+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe18014420 con 0x7fbe280f5410 2024-02-13T03:38:12.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.273+0000 7fbe1e7fc700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 0x7fbe14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:12.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.273+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe28059100 con 0x7fbe280f5410 2024-02-13T03:38:12.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.273+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe18040e10 con 0x7fbe280f5410 2024-02-13T03:38:12.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.273+0000 7fbe2d77e700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 0x7fbe14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:12.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.274+0000 7fbe2d77e700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 0x7fbe14032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbe240096a0 tx=0x7fbe24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:12.278 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.278+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe180672b0 con 0x7fbe280f5410 2024-02-13T03:38:12.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.486+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe280fafa0 con 0x7fbe280f5410 2024-02-13T03:38:12.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.487+0000 7fbe1e7fc700 1 -- 172.21.15.112:0/3292408536 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe1801c910 con 0x7fbe280f5410 2024-02-13T03:38:12.488 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:12.488 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 msgr2=0x7fbe14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 0x7fbe14032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbe240096a0 tx=0x7fbe24006b40).stop 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 msgr2=0x7fbe28109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbe180089f0 tx=0x7fbe1800b040).stop 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 shutdown_connections 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fbe1402ff60 0x7fbe14032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 --2- 172.21.15.112:0/3292408536 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbe280f5410 0x7fbe28109500 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 >> 172.21.15.112:0/3292408536 conn(0x7fbe280f0b70 msgr2=0x7fbe280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:12.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 shutdown_connections 2024-02-13T03:38:12.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:12.489+0000 7fbe301e3700 1 -- 172.21.15.112:0/3292408536 wait complete. 2024-02-13T03:38:12.491 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:11.035123+0000 mgr.smithi080.vyzlfg (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:12.488401+0000 mon.smithi080 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.112:0/3292408536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:14.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:14.102 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:14.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:13.035476+0000 mgr.smithi080.vyzlfg (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 -- 172.21.15.112:0/2283470836 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 msgr2=0x7fb4dc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 --2- 172.21.15.112:0/2283470836 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f90f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb4cc009fb0 tx=0x7fb4cc009b30).stop 2024-02-13T03:38:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 -- 172.21.15.112:0/2283470836 shutdown_connections 2024-02-13T03:38:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 --2- 172.21.15.112:0/2283470836 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f90f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 -- 172.21.15.112:0/2283470836 >> 172.21.15.112:0/2283470836 conn(0x7fb4dc0f0b70 msgr2=0x7fb4dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 -- 172.21.15.112:0/2283470836 shutdown_connections 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.043+0000 7fb4e448f700 1 -- 172.21.15.112:0/2283470836 wait complete. 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e448f700 1 Processor -- start 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e448f700 1 -- start start 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e448f700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e448f700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0f4ff0 con 0x7fb4dc060280 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e222b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e222b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43994/0 (socket says 172.21.15.112:43994) 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.044+0000 7fb4e222b700 1 -- 172.21.15.112:0/877622495 learned_addr learned my addr 172.21.15.112:0/877622495 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4e222b700 1 -- 172.21.15.112:0/877622495 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc005040 con 0x7fb4dc060280 2024-02-13T03:38:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4e222b700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb4cc0089f0 tx=0x7fb4cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4cc009320 con 0x7fb4dc060280 2024-02-13T03:38:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0f5190 con 0x7fb4dc060280 2024-02-13T03:38:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4cc013380 con 0x7fb4dc060280 2024-02-13T03:38:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4cc014e20 con 0x7fb4dc060280 2024-02-13T03:38:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.045+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc0f5620 con 0x7fb4dc060280 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.046+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4cc0095f0 con 0x7fb4dc060280 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.046+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc059100 con 0x7fb4dc060280 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.046+0000 7fb4d2ffd700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 0x7fb4c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.046+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4cc040e90 con 0x7fb4dc060280 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.046+0000 7fb4e1a2a700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 0x7fb4c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.047+0000 7fb4e1a2a700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 0x7fb4c80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb4d80096a0 tx=0x7fb4d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.052+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4cc014420 con 0x7fb4dc060280 2024-02-13T03:38:16.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.256+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4dc0fc090 con 0x7fb4dc060280 2024-02-13T03:38:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.256+0000 7fb4d2ffd700 1 -- 172.21.15.112:0/877622495 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4cc01d980 con 0x7fb4dc060280 2024-02-13T03:38:16.257 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:16.257 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 msgr2=0x7fb4c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 0x7fb4c80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb4d80096a0 tx=0x7fb4d8006b40).stop 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 msgr2=0x7fb4dc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb4cc0089f0 tx=0x7fb4cc00b040).stop 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 shutdown_connections 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb4c802ff10 0x7fb4c80323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 --2- 172.21.15.112:0/877622495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4dc060280 0x7fb4dc0f6b70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 >> 172.21.15.112:0/877622495 conn(0x7fb4dc0f0b70 msgr2=0x7fb4dc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 shutdown_connections 2024-02-13T03:38:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:16.258+0000 7fb4e448f700 1 -- 172.21.15.112:0/877622495 wait complete. 2024-02-13T03:38:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:15.035852+0000 mgr.smithi080.vyzlfg (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:16.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:16 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:16.257815+0000 mon.smithi080 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.112:0/877622495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:17.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:17.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:18.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:18 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:17.036233+0000 mgr.smithi080.vyzlfg (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 -- 172.21.15.112:0/3379560523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 msgr2=0x7f59bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 --2- 172.21.15.112:0/3379560523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f59ac009fb0 tx=0x7f59ac009b30).stop 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 -- 172.21.15.112:0/3379560523 shutdown_connections 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 --2- 172.21.15.112:0/3379560523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 -- 172.21.15.112:0/3379560523 >> 172.21.15.112:0/3379560523 conn(0x7f59bc0f0b70 msgr2=0x7f59bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 -- 172.21.15.112:0/3379560523 shutdown_connections 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.946+0000 7f59c2237700 1 -- 172.21.15.112:0/3379560523 wait complete. 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59c2237700 1 Processor -- start 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59c2237700 1 -- start start 2024-02-13T03:38:19.952 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59c2237700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59c2237700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0fb960 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59bb7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:44004/0 (socket says 172.21.15.112:44004) 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.947+0000 7f59bb7fe700 1 -- 172.21.15.112:0/1796603544 learned_addr learned my addr 172.21.15.112:0/1796603544 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59bb7fe700 1 -- 172.21.15.112:0/1796603544 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac005040 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59bb7fe700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f59ac005010 tx=0x7f59ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac009320 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59bc0fbb00 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59ac013380 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac014e50 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.948+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59bc0fbf90 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.949+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59ac014420 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.949+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc059100 con 0x7f59bc0f5410 2024-02-13T03:38:19.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.949+0000 7f59b3fff700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:19.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.949+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59ac040f70 con 0x7f59bc0f5410 2024-02-13T03:38:19.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.949+0000 7f59baffd700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:19.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.950+0000 7f59baffd700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 0x7f59a80323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f59a40096a0 tx=0x7f59a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:19.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:19.954+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59ac0095f0 con 0x7f59bc0f5410 2024-02-13T03:38:20.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.160+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59bc0fafa0 con 0x7f59bc0f5410 2024-02-13T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.161+0000 7f59b3fff700 1 -- 172.21.15.112:0/1796603544 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59ac01db70 con 0x7f59bc0f5410 2024-02-13T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:20.161 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 msgr2=0x7f59a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 0x7f59a80323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f59a40096a0 tx=0x7f59a4006b40).stop 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 msgr2=0x7f59bc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f59ac005010 tx=0x7f59ac00b040).stop 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 shutdown_connections 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:20.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 --2- 172.21.15.112:0/1796603544 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f59bc0f5410 0x7f59bc0fd4e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 >> 172.21.15.112:0/1796603544 conn(0x7f59bc0f0b70 msgr2=0x7f59bc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 shutdown_connections 2024-02-13T03:38:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:20.162+0000 7f59c2237700 1 -- 172.21.15.112:0/1796603544 wait complete. 2024-02-13T03:38:20.163 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:20.944 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:19.036553+0000 mgr.smithi080.vyzlfg (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:20.945 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:20 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:20.161098+0000 mon.smithi080 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.112:0/1796603544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:21.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:21.739 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:22.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:22 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:21.036889+0000 mgr.smithi080.vyzlfg (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:23.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/562814271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 msgr2=0x7f7c580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:23.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/562814271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c580f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f7c40009fb0 tx=0x7f7c40009b30).stop 2024-02-13T03:38:23.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/562814271 shutdown_connections 2024-02-13T03:38:23.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/562814271 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c580f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/562814271 >> 172.21.15.112:0/562814271 conn(0x7f7c580f09a0 msgr2=0x7f7c580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/562814271 shutdown_connections 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.682+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/562814271 wait complete. 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c5c9ef700 1 Processor -- start 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c5c9ef700 1 -- start start 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c5c9ef700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c5c9ef700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c581059c0 con 0x7f7c580f51a0 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:23.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c577fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54378/0 (socket says 172.21.15.112:54378) 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c577fe700 1 -- 172.21.15.112:0/1711310686 learned_addr learned my addr 172.21.15.112:0/1711310686 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.683+0000 7f7c577fe700 1 -- 172.21.15.112:0/1711310686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40005040 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c577fe700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7c400089f0 tx=0x7f7c4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c40009320 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c58105bc0 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c40013380 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c40014e20 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.684+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c58106050 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c400095f0 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c58045ad0 con 0x7f7c580f51a0 2024-02-13T03:38:23.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c4ffff700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 0x7f7c440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:23.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c40040e90 con 0x7f7c580f51a0 2024-02-13T03:38:23.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c56ffd700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 0x7f7c440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:23.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.685+0000 7f7c56ffd700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 0x7f7c440323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7c480096a0 tx=0x7f7c48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:23.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.689+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c40014420 con 0x7f7c580f51a0 2024-02-13T03:38:23.893 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.894+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c58106f00 con 0x7f7c580f51a0 2024-02-13T03:38:23.894 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.894+0000 7f7c4ffff700 1 -- 172.21.15.112:0/1711310686 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c4001d980 con 0x7f7c580f51a0 2024-02-13T03:38:23.894 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:23.894 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:23.895 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 msgr2=0x7f7c440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 0x7f7c440323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7c480096a0 tx=0x7f7c48006b40).stop 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 msgr2=0x7f7c581054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7c400089f0 tx=0x7f7c4000b040).stop 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 shutdown_connections 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7c4402ff10 0x7f7c440323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 --2- 172.21.15.112:0/1711310686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7c580f51a0 0x7f7c581054d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 >> 172.21.15.112:0/1711310686 conn(0x7f7c580f09a0 msgr2=0x7f7c580f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 shutdown_connections 2024-02-13T03:38:23.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:23.896+0000 7f7c5c9ef700 1 -- 172.21.15.112:0/1711310686 wait complete. 2024-02-13T03:38:23.899 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:23.037286+0000 mgr.smithi080.vyzlfg (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:24.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:24 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:23.894706+0000 mon.smithi080 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.112:0/1711310686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:25.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:25.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:26.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:26 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:25.037611+0000 mgr.smithi080.vyzlfg (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:27.685 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 -- 172.21.15.112:0/422534659 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 msgr2=0x7f04a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 --2- 172.21.15.112:0/422534659 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a80f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f0498009fb0 tx=0x7f0498009b30).stop 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 -- 172.21.15.112:0/422534659 shutdown_connections 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 --2- 172.21.15.112:0/422534659 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a80f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 -- 172.21.15.112:0/422534659 >> 172.21.15.112:0/422534659 conn(0x7f04a80f1c70 msgr2=0x7f04a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 -- 172.21.15.112:0/422534659 shutdown_connections 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.683+0000 7f04afb07700 1 -- 172.21.15.112:0/422534659 wait complete. 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04afb07700 1 Processor -- start 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04afb07700 1 -- start start 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04afb07700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04afb07700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a8106cd0 con 0x7f04a80f6510 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04ad8a3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04ad8a3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54406/0 (socket says 172.21.15.112:54406) 2024-02-13T03:38:27.686 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04ad8a3700 1 -- 172.21.15.112:0/1328501180 learned_addr learned my addr 172.21.15.112:0/1328501180 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.684+0000 7f04ad8a3700 1 -- 172.21.15.112:0/1328501180 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0498005040 con 0x7f04a80f6510 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f04ad8a3700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0498005010 tx=0x7f049800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0498009320 con 0x7f04a80f6510 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a8106ed0 con 0x7f04a80f6510 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0498013380 con 0x7f04a80f6510 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0498014e50 con 0x7f04a80f6510 2024-02-13T03:38:27.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.685+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a8107360 con 0x7f04a80f6510 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0498014420 con 0x7f04a80f6510 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a8045ad0 con 0x7f04a80f6510 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f049e7fc700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 0x7f04940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0498040e30 con 0x7f04a80f6510 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f04ad0a2700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 0x7f04940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:27.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.686+0000 7f04ad0a2700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 0x7f04940323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f04a40096a0 tx=0x7f04a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:27.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.690+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04980095f0 con 0x7f04a80f6510 2024-02-13T03:38:27.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.896+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04a8108210 con 0x7f04a80f6510 2024-02-13T03:38:27.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.896+0000 7f049e7fc700 1 -- 172.21.15.112:0/1328501180 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f049801db70 con 0x7f04a80f6510 2024-02-13T03:38:27.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:27.897 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 msgr2=0x7f04940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 0x7f04940323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f04a40096a0 tx=0x7f04a4006b40).stop 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 msgr2=0x7f04a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0498005010 tx=0x7f049800b040).stop 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 shutdown_connections 2024-02-13T03:38:27.899 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f049402ff10 0x7f04940323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:27.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 --2- 172.21.15.112:0/1328501180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f04a80f6510 0x7f04a81067e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:27.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 >> 172.21.15.112:0/1328501180 conn(0x7f04a80f1c70 msgr2=0x7f04a80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:27.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 shutdown_connections 2024-02-13T03:38:27.900 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:27.898+0000 7f04afb07700 1 -- 172.21.15.112:0/1328501180 wait complete. 2024-02-13T03:38:27.900 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:27.037869+0000 mgr.smithi080.vyzlfg (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:28.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:28 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:27.897925+0000 mon.smithi080 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.112:0/1328501180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:29.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:29.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:30.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:30 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:29.038251+0000 mgr.smithi080.vyzlfg (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- 172.21.15.112:0/16491247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 msgr2=0x7f01f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 --2- 172.21.15.112:0/16491247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f40f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f01e4009fb0 tx=0x7f01e4009b30).stop 2024-02-13T03:38:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- 172.21.15.112:0/16491247 shutdown_connections 2024-02-13T03:38:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 --2- 172.21.15.112:0/16491247 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f40f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- 172.21.15.112:0/16491247 >> 172.21.15.112:0/16491247 conn(0x7f01f40f0b70 msgr2=0x7f01f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- 172.21.15.112:0/16491247 shutdown_connections 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- 172.21.15.112:0/16491247 wait complete. 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 Processor -- start 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.436+0000 7f01f8c07700 1 -- start start 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f8c07700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f8c07700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f4105c10 con 0x7f01f40f5410 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37342/0 (socket says 172.21.15.112:37342) 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f259c700 1 -- 172.21.15.112:0/1007601352 learned_addr learned my addr 172.21.15.112:0/1007601352 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f259c700 1 -- 172.21.15.112:0/1007601352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4005040 con 0x7f01f40f5410 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.437+0000 7f01f259c700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f01e4005010 tx=0x7f01e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e4009320 con 0x7f01f40f5410 2024-02-13T03:38:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f4105e10 con 0x7f01f40f5410 2024-02-13T03:38:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e4013380 con 0x7f01f40f5410 2024-02-13T03:38:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e4014e50 con 0x7f01f40f5410 2024-02-13T03:38:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f41062a0 con 0x7f01f40f5410 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01e4014420 con 0x7f01f40f5410 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.438+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f4045ad0 con 0x7f01f40f5410 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.439+0000 7f01e2ffd700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.439+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01e4040e30 con 0x7f01f40f5410 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.439+0000 7f01f1d9b700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.439+0000 7f01f1d9b700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 0x7f01dc0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f01e80096a0 tx=0x7f01e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:31.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.443+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01e40095f0 con 0x7f01f40f5410 2024-02-13T03:38:31.649 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.649+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01f4107150 con 0x7f01f40f5410 2024-02-13T03:38:31.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.650+0000 7f01e2ffd700 1 -- 172.21.15.112:0/1007601352 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01e401db70 con 0x7f01f40f5410 2024-02-13T03:38:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:31.650 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:31.651 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 msgr2=0x7f01dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 0x7f01dc0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f01e80096a0 tx=0x7f01e8006b40).stop 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 msgr2=0x7f01f4105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f01e4005010 tx=0x7f01e400b040).stop 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 shutdown_connections 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 --2- 172.21.15.112:0/1007601352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f01f40f5410 0x7f01f4105720 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 >> 172.21.15.112:0/1007601352 conn(0x7f01f40f0b70 msgr2=0x7f01f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 shutdown_connections 2024-02-13T03:38:31.652 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:31.651+0000 7f01f8c07700 1 -- 172.21.15.112:0/1007601352 wait complete. 2024-02-13T03:38:31.653 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:31.038522+0000 mgr.smithi080.vyzlfg (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:32.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:32 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:31.650936+0000 mon.smithi080 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.112:0/1007601352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:33.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:33.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:34.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:34 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:33.038819+0000 mgr.smithi080.vyzlfg (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- 172.21.15.112:0/1532367495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 msgr2=0x7f75380f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 --2- 172.21.15.112:0/1532367495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75380f4be0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f7528009fb0 tx=0x7f7528009b30).stop 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- 172.21.15.112:0/1532367495 shutdown_connections 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 --2- 172.21.15.112:0/1532367495 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75380f4be0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- 172.21.15.112:0/1532367495 >> 172.21.15.112:0/1532367495 conn(0x7f75380f02e0 msgr2=0x7f75380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- 172.21.15.112:0/1532367495 shutdown_connections 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- 172.21.15.112:0/1532367495 wait complete. 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 Processor -- start 2024-02-13T03:38:35.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.457+0000 7f75402a5700 1 -- start start 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f75402a5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f75402a5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7538106cc0 con 0x7f75380f47c0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f753e041700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f753e041700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37352/0 (socket says 172.21.15.112:37352) 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f753e041700 1 -- 172.21.15.112:0/530512298 learned_addr learned my addr 172.21.15.112:0/530512298 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f753e041700 1 -- 172.21.15.112:0/530512298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7528005040 con 0x7f75380f47c0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.458+0000 7f753e041700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7528005010 tx=0x7f752800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.459+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7528009320 con 0x7f75380f47c0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.459+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7538106ec0 con 0x7f75380f47c0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.459+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7528013380 con 0x7f75380f47c0 2024-02-13T03:38:35.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.459+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7528014e50 con 0x7f75380f47c0 2024-02-13T03:38:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.459+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7538107350 con 0x7f75380f47c0 2024-02-13T03:38:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75280095f0 con 0x7f75380f47c0 2024-02-13T03:38:35.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7538045ad0 con 0x7f75380f47c0 2024-02-13T03:38:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f752effd700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 0x7f75240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7528040e90 con 0x7f75380f47c0 2024-02-13T03:38:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f753d840700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 0x7f75240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:35.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.460+0000 7f753d840700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 0x7f75240323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f75340096a0 tx=0x7f7534006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:35.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.464+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7528014420 con 0x7f75380f47c0 2024-02-13T03:38:35.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.674+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7538108200 con 0x7f75380f47c0 2024-02-13T03:38:35.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.675+0000 7f752effd700 1 -- 172.21.15.112:0/530512298 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7528010230 con 0x7f75380f47c0 2024-02-13T03:38:35.675 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:35.675 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 msgr2=0x7f75240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 0x7f75240323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f75340096a0 tx=0x7f7534006b40).stop 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 msgr2=0x7f75381067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7528005010 tx=0x7f752800b040).stop 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 shutdown_connections 2024-02-13T03:38:35.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f752402ff10 0x7f75240323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:35.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 --2- 172.21.15.112:0/530512298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f75380f47c0 0x7f75381067d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:35.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 >> 172.21.15.112:0/530512298 conn(0x7f75380f02e0 msgr2=0x7f75380f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:35.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 shutdown_connections 2024-02-13T03:38:35.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:35.676+0000 7f75402a5700 1 -- 172.21.15.112:0/530512298 wait complete. 2024-02-13T03:38:35.678 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:35.039113+0000 mgr.smithi080.vyzlfg (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:36.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:36 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:35.675856+0000 mon.smithi080 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.112:0/530512298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:37.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:37.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:38.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:38 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:37.039429+0000 mgr.smithi080.vyzlfg (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 -- 172.21.15.112:0/2137011417 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 msgr2=0x7f895c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 --2- 172.21.15.112:0/2137011417 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f8950009fb0 tx=0x7f8950009b30).stop 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 -- 172.21.15.112:0/2137011417 shutdown_connections 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 --2- 172.21.15.112:0/2137011417 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 -- 172.21.15.112:0/2137011417 >> 172.21.15.112:0/2137011417 conn(0x7f895c0f0b70 msgr2=0x7f895c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 -- 172.21.15.112:0/2137011417 shutdown_connections 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.047+0000 7f89615a4700 1 -- 172.21.15.112:0/2137011417 wait complete. 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f89615a4700 1 Processor -- start 2024-02-13T03:38:39.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f89615a4700 1 -- start start 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f89615a4700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f89615a4700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c1072f0 con 0x7f895c0f5410 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f895affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f895affd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:37374/0 (socket says 172.21.15.112:37374) 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f895affd700 1 -- 172.21.15.112:0/3647017794 learned_addr learned my addr 172.21.15.112:0/3647017794 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.048+0000 7f895affd700 1 -- 172.21.15.112:0/3647017794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8950005040 con 0x7f895c0f5410 2024-02-13T03:38:39.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f895affd700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8950005010 tx=0x7f895000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:39.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8950009320 con 0x7f895c0f5410 2024-02-13T03:38:39.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895c060220 con 0x7f895c0f5410 2024-02-13T03:38:39.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8950013380 con 0x7f895c0f5410 2024-02-13T03:38:39.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8950014e50 con 0x7f895c0f5410 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.049+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895c10bdf0 con 0x7f895c0f5410 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89500263a0 con 0x7f895c0f5410 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895c045ad0 con 0x7f895c0f5410 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f89437fe700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 0x7f89440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8950041250 con 0x7f895c0f5410 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f895a7fc700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 0x7f89440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:39.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.050+0000 7f895a7fc700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 0x7f89440323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f894c0096a0 tx=0x7f894c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:39.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.054+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f895001ccf0 con 0x7f895c0f5410 2024-02-13T03:38:39.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.259+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f895c10c5b0 con 0x7f895c0f5410 2024-02-13T03:38:39.260 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.259+0000 7f89437fe700 1 -- 172.21.15.112:0/3647017794 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f895001cb10 con 0x7f895c0f5410 2024-02-13T03:38:39.260 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:39.260 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:39.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 msgr2=0x7f89440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:39.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 0x7f89440323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f894c0096a0 tx=0x7f894c006b40).stop 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 msgr2=0x7f895c10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8950005010 tx=0x7f895000b040).stop 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 shutdown_connections 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f894402ff10 0x7f89440323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 --2- 172.21.15.112:0/3647017794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f895c0f5410 0x7f895c10b900 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 >> 172.21.15.112:0/3647017794 conn(0x7f895c0f0b70 msgr2=0x7f895c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 shutdown_connections 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:39.261+0000 7f89615a4700 1 -- 172.21.15.112:0/3647017794 wait complete. 2024-02-13T03:38:39.262 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:39.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:39 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:39.260737+0000 mon.smithi080 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.112:0/3647017794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:40.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:40 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:39.039694+0000 mgr.smithi080.vyzlfg (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:41.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:41.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:42.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:42 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:41.039974+0000 mgr.smithi080.vyzlfg (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 -- 172.21.15.112:0/3478965954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 msgr2=0x7f645c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 --2- 172.21.15.112:0/3478965954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f6444009fb0 tx=0x7f6444009b30).stop 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 -- 172.21.15.112:0/3478965954 shutdown_connections 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 --2- 172.21.15.112:0/3478965954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 -- 172.21.15.112:0/3478965954 >> 172.21.15.112:0/3478965954 conn(0x7f645c0f0b70 msgr2=0x7f645c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:42.995 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 -- 172.21.15.112:0/3478965954 shutdown_connections 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.970+0000 7f6460c84700 1 -- 172.21.15.112:0/3478965954 wait complete. 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6460c84700 1 Processor -- start 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6460c84700 1 -- start start 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6460c84700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6460c84700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c060240 con 0x7f645c0f5410 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f645a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f645a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47302/0 (socket says 172.21.15.112:47302) 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f645a59c700 1 -- 172.21.15.112:0/2815147441 learned_addr learned my addr 172.21.15.112:0/2815147441 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f645a59c700 1 -- 172.21.15.112:0/2815147441 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6444005040 con 0x7f645c0f5410 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f645a59c700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f6444005010 tx=0x7f644400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6444009320 con 0x7f645c0f5410 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.971+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c060440 con 0x7f645c0f5410 2024-02-13T03:38:42.996 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6444013380 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6444014e50 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c109b90 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64440095f0 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c045ad0 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6452ffd700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 0x7f64480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.972+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6444040e90 con 0x7f645c0f5410 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.973+0000 7f6459d9b700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 0x7f64480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.973+0000 7f6459d9b700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 0x7f64480323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f644c0096a0 tx=0x7f644c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:42.997 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:42.977+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6444014420 con 0x7f645c0f5410 2024-02-13T03:38:43.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.181+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f645c10a3b0 con 0x7f645c0f5410 2024-02-13T03:38:43.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.182+0000 7f6452ffd700 1 -- 172.21.15.112:0/2815147441 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6444010230 con 0x7f645c0f5410 2024-02-13T03:38:43.183 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:43.183 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:43.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 msgr2=0x7f64480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:43.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 0x7f64480323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f644c0096a0 tx=0x7f644c006b40).stop 2024-02-13T03:38:43.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 msgr2=0x7f645c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:43.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f6444005010 tx=0x7f644400b040).stop 2024-02-13T03:38:43.184 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 shutdown_connections 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f644802ff10 0x7f64480323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 --2- 172.21.15.112:0/2815147441 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f645c0f5410 0x7f645c1094e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 >> 172.21.15.112:0/2815147441 conn(0x7f645c0f0b70 msgr2=0x7f645c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 shutdown_connections 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:43.184+0000 7f6460c84700 1 -- 172.21.15.112:0/2815147441 wait complete. 2024-02-13T03:38:43.185 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:43.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:43 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:43.183353+0000 mon.smithi080 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.112:0/2815147441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:44.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:44.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:44.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:44 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:43.040336+0000 mgr.smithi080.vyzlfg (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 -- 172.21.15.112:0/193347057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 msgr2=0x7f7a1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 --2- 172.21.15.112:0/193347057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c0f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f7a14009fb0 tx=0x7f7a14009b30).stop 2024-02-13T03:38:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 -- 172.21.15.112:0/193347057 shutdown_connections 2024-02-13T03:38:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 --2- 172.21.15.112:0/193347057 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c0f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:46.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 -- 172.21.15.112:0/193347057 >> 172.21.15.112:0/193347057 conn(0x7f7a1c0f0b70 msgr2=0x7f7a1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:46.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.828+0000 7f7a20c79700 1 -- 172.21.15.112:0/193347057 shutdown_connections 2024-02-13T03:38:46.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.829+0000 7f7a20c79700 1 -- 172.21.15.112:0/193347057 wait complete. 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.829+0000 7f7a20c79700 1 Processor -- start 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.829+0000 7f7a20c79700 1 -- start start 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.829+0000 7f7a20c79700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.829+0000 7f7a20c79700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c060220 con 0x7f7a1c0f5410 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a1a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a1a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47328/0 (socket says 172.21.15.112:47328) 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a1a59c700 1 -- 172.21.15.112:0/479017257 learned_addr learned my addr 172.21.15.112:0/479017257 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a1a59c700 1 -- 172.21.15.112:0/479017257 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14005040 con 0x7f7a1c0f5410 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a1a59c700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f7a140089f0 tx=0x7f7a1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a14009320 con 0x7f7a1c0f5410 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c060420 con 0x7f7a1c0f5410 2024-02-13T03:38:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a14013380 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1401ed80 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.830+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c109bb0 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.831+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c059100 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.831+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a1401e420 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.831+0000 7f7a12ffd700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 0x7f7a08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.831+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a14041060 con 0x7f7a1c0f5410 2024-02-13T03:38:46.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.831+0000 7f7a19d9b700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 0x7f7a08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:46.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.832+0000 7f7a19d9b700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 0x7f7a08032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7a040096a0 tx=0x7f7a04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:46.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:46.836+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a1401cb10 con 0x7f7a1c0f5410 2024-02-13T03:38:46.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:46 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:45.040728+0000 mgr.smithi080.vyzlfg (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:47.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.041+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a1c045ad0 con 0x7f7a1c0f5410 2024-02-13T03:38:47.042 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.042+0000 7f7a12ffd700 1 -- 172.21.15.112:0/479017257 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a1402c2d0 con 0x7f7a1c0f5410 2024-02-13T03:38:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:47.043 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 msgr2=0x7f7a08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 0x7f7a08032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7a040096a0 tx=0x7f7a04006b40).stop 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 msgr2=0x7f7a1c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f7a140089f0 tx=0x7f7a1400b040).stop 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 shutdown_connections 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7a0802ff60 0x7f7a08032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 --2- 172.21.15.112:0/479017257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a1c0f5410 0x7f7a1c109500 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 >> 172.21.15.112:0/479017257 conn(0x7f7a1c0f0b70 msgr2=0x7f7a1c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 shutdown_connections 2024-02-13T03:38:47.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:47.043+0000 7f7a20c79700 1 -- 172.21.15.112:0/479017257 wait complete. 2024-02-13T03:38:47.045 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:47.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:47 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:47.043239+0000 mon.smithi080 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.112:0/479017257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:48.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:48.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:48.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:48 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:47.041166+0000 mgr.smithi080.vyzlfg (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.636+0000 7fa6f1589700 1 -- 172.21.15.112:0/4045200192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 msgr2=0x7fa6ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.636+0000 7fa6f1589700 1 --2- 172.21.15.112:0/4045200192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fa6e0009fb0 tx=0x7fa6e0009b30).stop 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.636+0000 7fa6f1589700 1 -- 172.21.15.112:0/4045200192 shutdown_connections 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.636+0000 7fa6f1589700 1 --2- 172.21.15.112:0/4045200192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.636+0000 7fa6f1589700 1 -- 172.21.15.112:0/4045200192 >> 172.21.15.112:0/4045200192 conn(0x7fa6ec0f0b70 msgr2=0x7fa6ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 -- 172.21.15.112:0/4045200192 shutdown_connections 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 -- 172.21.15.112:0/4045200192 wait complete. 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 Processor -- start 2024-02-13T03:38:50.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 -- start start 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.637+0000 7fa6f1589700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec060240 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6eaffd700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:47340/0 (socket says 172.21.15.112:47340) 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6eaffd700 1 -- 172.21.15.112:0/3603068827 learned_addr learned my addr 172.21.15.112:0/3603068827 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6eaffd700 1 -- 172.21.15.112:0/3603068827 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0005040 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6eaffd700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa6e0005010 tx=0x7fa6e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e0009320 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6e0013380 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec060440 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e0014e50 con 0x7fa6ec0f5410 2024-02-13T03:38:50.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.638+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec109bd0 con 0x7fa6ec0f5410 2024-02-13T03:38:50.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.639+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6e00095f0 con 0x7fa6ec0f5410 2024-02-13T03:38:50.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.639+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec045ad0 con 0x7fa6ec0f5410 2024-02-13T03:38:50.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.639+0000 7fa6d37fe700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 0x7fa6d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:50.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.639+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6e0040e90 con 0x7fa6ec0f5410 2024-02-13T03:38:50.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.639+0000 7fa6ea7fc700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 0x7fa6d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:50.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.640+0000 7fa6ea7fc700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 0x7fa6d40323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa6dc0096a0 tx=0x7fa6dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:50.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.644+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6e0014420 con 0x7fa6ec0f5410 2024-02-13T03:38:50.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.847+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6ec10a3f0 con 0x7fa6ec0f5410 2024-02-13T03:38:50.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.848+0000 7fa6d37fe700 1 -- 172.21.15.112:0/3603068827 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6e0010230 con 0x7fa6ec0f5410 2024-02-13T03:38:50.848 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:50.849 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 msgr2=0x7fa6d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 0x7fa6d40323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa6dc0096a0 tx=0x7fa6dc006b40).stop 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 msgr2=0x7fa6ec109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa6e0005010 tx=0x7fa6e000b040).stop 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 shutdown_connections 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fa6d402ff10 0x7fa6d40323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 --2- 172.21.15.112:0/3603068827 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6ec0f5410 0x7fa6ec109520 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 >> 172.21.15.112:0/3603068827 conn(0x7fa6ec0f0b70 msgr2=0x7fa6ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:50.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 shutdown_connections 2024-02-13T03:38:50.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:50.849+0000 7fa6f1589700 1 -- 172.21.15.112:0/3603068827 wait complete. 2024-02-13T03:38:50.851 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:50.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:50 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:49.041469+0000 mgr.smithi080.vyzlfg (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:51.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:51 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:50.849089+0000 mon.smithi080 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.112:0/3603068827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:52.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:52.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:52.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:52 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:51.041842+0000 mgr.smithi080.vyzlfg (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:54.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 -- 172.21.15.112:0/2714917758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 msgr2=0x7ff4a40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:54.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/2714917758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f90f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7ff498009fb0 tx=0x7ff498009b30).stop 2024-02-13T03:38:54.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 -- 172.21.15.112:0/2714917758 shutdown_connections 2024-02-13T03:38:54.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/2714917758 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f90f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:54.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 -- 172.21.15.112:0/2714917758 >> 172.21.15.112:0/2714917758 conn(0x7ff4a40f0b70 msgr2=0x7ff4a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:54.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 -- 172.21.15.112:0/2714917758 shutdown_connections 2024-02-13T03:38:54.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.643+0000 7ff4aaf74700 1 -- 172.21.15.112:0/2714917758 wait complete. 2024-02-13T03:38:54.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4aaf74700 1 Processor -- start 2024-02-13T03:38:54.644 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4aaf74700 1 -- start start 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4aaf74700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4aaf74700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40f4ff0 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4a8d10700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4a8d10700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54970/0 (socket says 172.21.15.112:54970) 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.644+0000 7ff4a8d10700 1 -- 172.21.15.112:0/3013408781 learned_addr learned my addr 172.21.15.112:0/3013408781 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4a8d10700 1 -- 172.21.15.112:0/3013408781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498005040 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4a8d10700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7ff4980089f0 tx=0x7ff49800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff498009320 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a40f5190 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff498013380 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff498014e20 con 0x7ff4a4060280 2024-02-13T03:38:54.646 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.645+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a40f5620 con 0x7ff4a4060280 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff498014420 con 0x7ff4a4060280 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a4059100 con 0x7ff4a4060280 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4a17fa700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff498040f00 con 0x7ff4a4060280 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4a3fff700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:54.647 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.646+0000 7ff4a3fff700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 0x7ff4940323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff4900096a0 tx=0x7ff490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:54.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.651+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4980095f0 con 0x7ff4a4060280 2024-02-13T03:38:54.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.856+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4a40fc090 con 0x7ff4a4060280 2024-02-13T03:38:54.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.856+0000 7ff4a17fa700 1 -- 172.21.15.112:0/3013408781 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff49801db70 con 0x7ff4a4060280 2024-02-13T03:38:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:54.856 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:54.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 msgr2=0x7ff4940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:54.857 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 0x7ff4940323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff4900096a0 tx=0x7ff490006b40).stop 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 msgr2=0x7ff4a40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7ff4980089f0 tx=0x7ff49800b040).stop 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 shutdown_connections 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 --2- 172.21.15.112:0/3013408781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff4a4060280 0x7ff4a40f6b70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 >> 172.21.15.112:0/3013408781 conn(0x7ff4a40f0b70 msgr2=0x7ff4a40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 shutdown_connections 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:54.858+0000 7ff4aaf74700 1 -- 172.21.15.112:0/3013408781 wait complete. 2024-02-13T03:38:54.858 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:54.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:54 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:53.042107+0000 mgr.smithi080.vyzlfg (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:55.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:55 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:54.856821+0000 mon.smithi080 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.112:0/3013408781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:38:56.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:38:56.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:38:56.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:56 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:55.042383+0000 mgr.smithi080.vyzlfg (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:57.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:56.866284+0000 mon.smithi080 (mon.0) 335 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/mirror_snapshot_schedule"}]: dispatch 2024-02-13T03:38:57.941 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:57 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:56.872880+0000 mon.smithi080 (mon.0) 336 : audit [INF] from='mgr.14162 172.21.15.80:0/213827996' entity='mgr.smithi080.vyzlfg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi080.vyzlfg/trash_purge_schedule"}]: dispatch 2024-02-13T03:38:58.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 -- 172.21.15.112:0/1731858012 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 msgr2=0x7f949c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:58.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 --2- 172.21.15.112:0/1731858012 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c0f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f948c009fb0 tx=0x7f948c009b30).stop 2024-02-13T03:38:58.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 -- 172.21.15.112:0/1731858012 shutdown_connections 2024-02-13T03:38:58.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 --2- 172.21.15.112:0/1731858012 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c0f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 -- 172.21.15.112:0/1731858012 >> 172.21.15.112:0/1731858012 conn(0x7f949c0f0b70 msgr2=0x7f949c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 -- 172.21.15.112:0/1731858012 shutdown_connections 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 -- 172.21.15.112:0/1731858012 wait complete. 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.341+0000 7f94a3b77700 1 Processor -- start 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a3b77700 1 -- start start 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a3b77700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a3b77700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949c105bd0 con 0x7f949c0f5410 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a1913700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a1913700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:54980/0 (socket says 172.21.15.112:54980) 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a1913700 1 -- 172.21.15.112:0/4003065141 learned_addr learned my addr 172.21.15.112:0/4003065141 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a1913700 1 -- 172.21.15.112:0/4003065141 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c005040 con 0x7f949c0f5410 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a1913700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f948c005010 tx=0x7f948c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c009320 con 0x7f949c0f5410 2024-02-13T03:38:58.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949c105dd0 con 0x7f949c0f5410 2024-02-13T03:38:58.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f948c013380 con 0x7f949c0f5410 2024-02-13T03:38:58.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.342+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c014e50 con 0x7f949c0f5410 2024-02-13T03:38:58.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.343+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949c106260 con 0x7f949c0f5410 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.343+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f948c0263a0 con 0x7f949c0f5410 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.343+0000 7f94927fc700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 0x7f94880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.343+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949c045ad0 con 0x7f949c0f5410 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.343+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f948c040e60 con 0x7f949c0f5410 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.344+0000 7f94a1112700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 0x7f94880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:38:58.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.344+0000 7f94a1112700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 0x7f94880323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f94980096a0 tx=0x7f9498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:38:58.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.349+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f948c012920 con 0x7f949c0f5410 2024-02-13T03:38:58.552 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.553+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f949c0fafa0 con 0x7f949c0f5410 2024-02-13T03:38:58.553 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.553+0000 7f94927fc700 1 -- 172.21.15.112:0/4003065141 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f948c00f340 con 0x7f949c0f5410 2024-02-13T03:38:58.553 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:38:58.553 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:38:58.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 msgr2=0x7f94880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:58.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 0x7f94880323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f94980096a0 tx=0x7f9498006b40).stop 2024-02-13T03:38:58.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 msgr2=0x7f949c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:38:58.555 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f948c005010 tx=0x7f948c00b040).stop 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 shutdown_connections 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f948802ff10 0x7f94880323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 --2- 172.21.15.112:0/4003065141 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f949c0f5410 0x7f949c1056e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 >> 172.21.15.112:0/4003065141 conn(0x7f949c0f0b70 msgr2=0x7f949c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 shutdown_connections 2024-02-13T03:38:58.556 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:38:58.555+0000 7f94a3b77700 1 -- 172.21.15.112:0/4003065141 wait complete. 2024-02-13T03:38:58.557 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:38:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:57.042642+0000 mgr.smithi080.vyzlfg (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:38:58.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:38:58 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:38:58.553689+0000 mon.smithi080 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.112:0/4003065141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:39:00.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:39:00.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:39:00.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:00 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:38:59.042995+0000 mgr.smithi080.vyzlfg (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.327+0000 7f6943f29700 1 -- 172.21.15.112:0/1271827136 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 msgr2=0x7f693c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.327+0000 7f6943f29700 1 --2- 172.21.15.112:0/1271827136 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c0f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f6930009fb0 tx=0x7f6930009b30).stop 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.327+0000 7f6943f29700 1 -- 172.21.15.112:0/1271827136 shutdown_connections 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.327+0000 7f6943f29700 1 --2- 172.21.15.112:0/1271827136 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c0f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.327+0000 7f6943f29700 1 -- 172.21.15.112:0/1271827136 >> 172.21.15.112:0/1271827136 conn(0x7f693c0f0b70 msgr2=0x7f693c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 -- 172.21.15.112:0/1271827136 shutdown_connections 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 -- 172.21.15.112:0/1271827136 wait complete. 2024-02-13T03:39:02.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 Processor -- start 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 -- start start 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6943f29700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c106cc0 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6941cc5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6941cc5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52444/0 (socket says 172.21.15.112:52444) 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.328+0000 7f6941cc5700 1 -- 172.21.15.112:0/2352868421 learned_addr learned my addr 172.21.15.112:0/2352868421 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f6941cc5700 1 -- 172.21.15.112:0/2352868421 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930005040 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f6941cc5700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f6930005010 tx=0x7f693000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6930009320 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c106ec0 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6930013380 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c107350 con 0x7f693c0f5410 2024-02-13T03:39:02.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.329+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6930014dd0 con 0x7f693c0f5410 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.330+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69300095f0 con 0x7f693c0f5410 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.330+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c045ad0 con 0x7f693c0f5410 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.330+0000 7f692e7fc700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.330+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6930040e70 con 0x7f693c0f5410 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.330+0000 7f69414c4700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.331+0000 7f69414c4700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 0x7f6928032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f69380096a0 tx=0x7f6938006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:02.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.335+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6930014420 con 0x7f693c0f5410 2024-02-13T03:39:02.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.542+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f693c108200 con 0x7f693c0f5410 2024-02-13T03:39:02.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.543+0000 7f692e7fc700 1 -- 172.21.15.112:0/2352868421 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f693000e2a0 con 0x7f693c0f5410 2024-02-13T03:39:02.543 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:39:02.543 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:39:02.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 msgr2=0x7f6928032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:02.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 0x7f6928032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f69380096a0 tx=0x7f6938006b40).stop 2024-02-13T03:39:02.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 msgr2=0x7f693c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f6930005010 tx=0x7f693000b040).stop 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 shutdown_connections 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 --2- 172.21.15.112:0/2352868421 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f693c0f5410 0x7f693c1067d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 >> 172.21.15.112:0/2352868421 conn(0x7f693c0f0b70 msgr2=0x7f693c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 shutdown_connections 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:02.545+0000 7f6943f29700 1 -- 172.21.15.112:0/2352868421 wait complete. 2024-02-13T03:39:02.545 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:39:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:01.043391+0000 mgr.smithi080.vyzlfg (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:02.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:02 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:39:02.543575+0000 mon.smithi080 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.112:0/2352868421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:39:04.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:39:04.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:39:04.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:04 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:03.043723+0000 mgr.smithi080.vyzlfg (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:06.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 -- 172.21.15.112:0/173440268 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 msgr2=0x7f7d680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:06.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 --2- 172.21.15.112:0/173440268 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d680f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f7d5c009fb0 tx=0x7f7d5c009b30).stop 2024-02-13T03:39:06.178 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 -- 172.21.15.112:0/173440268 shutdown_connections 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 --2- 172.21.15.112:0/173440268 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d680f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 -- 172.21.15.112:0/173440268 >> 172.21.15.112:0/173440268 conn(0x7f7d680f0b70 msgr2=0x7f7d680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 -- 172.21.15.112:0/173440268 shutdown_connections 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.177+0000 7f7d6f834700 1 -- 172.21.15.112:0/173440268 wait complete. 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.178+0000 7f7d6f834700 1 Processor -- start 2024-02-13T03:39:06.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.178+0000 7f7d6f834700 1 -- start start 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.178+0000 7f7d6f834700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.178+0000 7f7d6f834700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68060220 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6d5d0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6d5d0700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52466/0 (socket says 172.21.15.112:52466) 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6d5d0700 1 -- 172.21.15.112:0/2255074686 learned_addr learned my addr 172.21.15.112:0/2255074686 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6d5d0700 1 -- 172.21.15.112:0/2255074686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c005040 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6d5d0700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f7d5c005010 tx=0x7f7d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c009320 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68060420 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d5c013380 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c014e50 con 0x7f7d680f5410 2024-02-13T03:39:06.181 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.179+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d68109bb0 con 0x7f7d680f5410 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.180+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d5c014420 con 0x7f7d680f5410 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.180+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68059100 con 0x7f7d680f5410 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.180+0000 7f7d59ffb700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 0x7f7d540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.180+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d5c040e30 con 0x7f7d680f5410 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.181+0000 7f7d6cdcf700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 0x7f7d540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:06.182 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.181+0000 7f7d6cdcf700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 0x7f7d540323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7d640096a0 tx=0x7f7d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:06.185 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.185+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d5c0095f0 con 0x7f7d680f5410 2024-02-13T03:39:06.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.393+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d68045ad0 con 0x7f7d680f5410 2024-02-13T03:39:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.393+0000 7f7d59ffb700 1 -- 172.21.15.112:0/2255074686 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d5c01db70 con 0x7f7d680f5410 2024-02-13T03:39:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:39:06.394 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 msgr2=0x7f7d540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 0x7f7d540323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7d640096a0 tx=0x7f7d64006b40).stop 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 msgr2=0x7f7d68109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f7d5c005010 tx=0x7f7d5c00b040).stop 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 shutdown_connections 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7f7d5402ff10 0x7f7d540323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 --2- 172.21.15.112:0/2255074686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d680f5410 0x7f7d68109500 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 >> 172.21.15.112:0/2255074686 conn(0x7f7d680f0b70 msgr2=0x7f7d680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 shutdown_connections 2024-02-13T03:39:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:06.395+0000 7f7d6f834700 1 -- 172.21.15.112:0/2255074686 wait complete. 2024-02-13T03:39:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:39:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:05.044054+0000 mgr.smithi080.vyzlfg (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:06.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:06 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:39:06.394715+0000 mon.smithi080 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.112:0/2255074686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:39:08.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:39:08.037 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:39:08.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:08 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:07.044455+0000 mgr.smithi080.vyzlfg (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:10.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/2416308538 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 msgr2=0x7fb6a40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/2416308538 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f4b60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fb68c009fb0 tx=0x7fb68c009b30).stop 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/2416308538 shutdown_connections 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/2416308538 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f4b60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/2416308538 >> 172.21.15.112:0/2416308538 conn(0x7fb6a40f0260 msgr2=0x7fb6a40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/2416308538 shutdown_connections 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.050+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/2416308538 wait complete. 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a8f2d700 1 Processor -- start 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a8f2d700 1 -- start start 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a8f2d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a8f2d700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a40f5770 con 0x7fb6a40f4740 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:52492/0 (socket says 172.21.15.112:52492) 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.051+0000 7fb6a259c700 1 -- 172.21.15.112:0/1710098901 learned_addr learned my addr 172.21.15.112:0/1710098901 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:39:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb6a259c700 1 -- 172.21.15.112:0/1710098901 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c005040 con 0x7fb6a40f4740 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb6a259c700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb68c005010 tx=0x7fb68c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb68c009320 con 0x7fb6a40f4740 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a40f5910 con 0x7fb6a40f4740 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb68c013380 con 0x7fb6a40f4740 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb68c014e50 con 0x7fb6a40f4740 2024-02-13T03:39:10.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.052+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a40f5da0 con 0x7fb6a40f4740 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.053+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb68c0095f0 con 0x7fb6a40f4740 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.053+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4059100 con 0x7fb6a40f4740 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.053+0000 7fb69affd700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 0x7fb6900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.053+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb68c040e60 con 0x7fb6a40f4740 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.053+0000 7fb6a1d9b700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 0x7fb6900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:10.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.054+0000 7fb6a1d9b700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 0x7fb6900323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb6940096a0 tx=0x7fb694006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:10.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.058+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb68c014420 con 0x7fb6a40f4740 2024-02-13T03:39:10.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.262+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6a40fc080 con 0x7fb6a40f4740 2024-02-13T03:39:10.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.263+0000 7fb69affd700 1 -- 172.21.15.112:0/1710098901 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb68c010230 con 0x7fb6a40f4740 2024-02-13T03:39:10.264 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:39:10.264 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:39:10.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 msgr2=0x7fb6900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:10.265 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 0x7fb6900323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb6940096a0 tx=0x7fb694006b40).stop 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 msgr2=0x7fb6a40f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb68c005010 tx=0x7fb68c00b040).stop 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 shutdown_connections 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fb69002ff10 0x7fb6900323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 --2- 172.21.15.112:0/1710098901 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb6a40f4740 0x7fb6a40f72f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 >> 172.21.15.112:0/1710098901 conn(0x7fb6a40f0260 msgr2=0x7fb6a40f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 shutdown_connections 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:10.265+0000 7fb6a8f2d700 1 -- 172.21.15.112:0/1710098901 wait complete. 2024-02-13T03:39:10.266 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:39:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:09.044758+0000 mgr.smithi080.vyzlfg (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:10.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:10 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:39:10.264412+0000 mon.smithi080 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.112:0/1710098901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:39:11.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-13T03:39:11.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T03:39:12.940 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:12 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:11.045235+0000 mgr.smithi080.vyzlfg (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 -- 172.21.15.112:0/637088635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 msgr2=0x7fba8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 --2- 172.21.15.112:0/637088635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fba80009fb0 tx=0x7fba80009b30).stop 2024-02-13T03:39:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 -- 172.21.15.112:0/637088635 shutdown_connections 2024-02-13T03:39:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 --2- 172.21.15.112:0/637088635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 -- 172.21.15.112:0/637088635 >> 172.21.15.112:0/637088635 conn(0x7fba8c0f0b70 msgr2=0x7fba8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 -- 172.21.15.112:0/637088635 shutdown_connections 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 -- 172.21.15.112:0/637088635 wait complete. 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.841+0000 7fba911ab700 1 Processor -- start 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba911ab700 1 -- start start 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba911ab700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba911ab700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c060240 con 0x7fba8c0f5410 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba8ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba8ad9d700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.112:43064/0 (socket says 172.21.15.112:43064) 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba8ad9d700 1 -- 172.21.15.112:0/1107072981 learned_addr learned my addr 172.21.15.112:0/1107072981 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba8ad9d700 1 -- 172.21.15.112:0/1107072981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80005040 con 0x7fba8c0f5410 2024-02-13T03:39:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba8ad9d700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fba80005010 tx=0x7fba8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba80009320 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8c060440 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.842+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba80013380 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba80014e50 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8c109b90 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba80014420 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8c045ad0 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba737fe700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 0x7fba740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.843+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba80040e60 con 0x7fba8c0f5410 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.844+0000 7fba8a59c700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 0x7fba740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.844+0000 7fba8a59c700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 0x7fba740323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fba7c0096a0 tx=0x7fba7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-13T03:39:13.852 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:13.848+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba800095f0 con 0x7fba8c0f5410 2024-02-13T03:39:14.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.051+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba8c10a3b0 con 0x7fba8c0f5410 2024-02-13T03:39:14.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.051+0000 7fba737fe700 1 -- 172.21.15.112:0/1107072981 <== mon.0 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba8001db70 con 0x7fba8c0f5410 2024-02-13T03:39:14.052 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-02-13T03:39:14.052 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":1,"fsid":"84391b2a-ca1f-11ee-95b9-87774f69a715","modified":"2024-02-13T03:25:47.662167Z","created":"2024-02-13T03:25:47.662167Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi080","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T03:39:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 msgr2=0x7fba740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 0x7fba740323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fba7c0096a0 tx=0x7fba7c006b40).stop 2024-02-13T03:39:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 msgr2=0x7fba8c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-13T03:39:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fba80005010 tx=0x7fba8000b040).stop 2024-02-13T03:39:14.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 shutdown_connections 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:6800/2233275020,v1:172.21.15.80:6801/2233275020] conn(0x7fba7402ff10 0x7fba740323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 --2- 172.21.15.112:0/1107072981 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fba8c0f5410 0x7fba8c1094e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 >> 172.21.15.112:0/1107072981 conn(0x7fba8c0f0b70 msgr2=0x7fba8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 shutdown_connections 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:2024-02-13T03:39:14.053+0000 7fba911ab700 1 -- 172.21.15.112:0/1107072981 wait complete. 2024-02-13T03:39:14.054 INFO:teuthology.orchestra.run.smithi112.stderr:dumped monmap epoch 1 2024-02-13T03:39:14.721 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-13T03:39:14.722 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-13T03:39:14.723 DEBUG:teuthology.orchestra.run.smithi080:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T03:39:14.747 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T03:39:14.766 INFO:tasks.cephadm:Stopping all daemons... 2024-02-13T03:39:14.766 INFO:tasks.cephadm.mon.smithi080:Stopping mon.smithi080... 2024-02-13T03:39:14.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080 2024-02-13T03:39:14.831 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: cluster 2024-02-13T03:39:13.045605+0000 mgr.smithi080.vyzlfg (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T03:39:14.831 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: audit 2024-02-13T03:39:14.052610+0000 mon.smithi080 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.112:0/1107072981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T03:39:15.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:14 smithi080 systemd[1]: Stopping Ceph mon.smithi080 for 84391b2a-ca1f-11ee-95b9-87774f69a715... 2024-02-13T03:39:15.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:39:14.957+0000 7ff22f500700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi080 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-13T03:39:15.190 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:14 smithi080 ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080[39409]: debug 2024-02-13T03:39:14.957+0000 7ff22f500700 -1 mon.smithi080@0(leader) e1 *** Got Signal Terminated *** 2024-02-13T03:39:15.690 INFO:journalctl@ceph.mon.smithi080.smithi080.stdout:Feb 13 03:39:15 smithi080 podman[45307]: ceph-84391b2a-ca1f-11ee-95b9-87774f69a715-mon.smithi080 2024-02-13T03:39:15.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi080.service' 2024-02-13T03:39:15.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T03:39:15.858 INFO:tasks.cephadm.mon.smithi080:Stopped mon.smithi080 2024-02-13T03:39:15.859 INFO:tasks.cephadm.mon.smithi112:Stopping mon.smithi112... 2024-02-13T03:39:15.859 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112 2024-02-13T03:39:15.896 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112.service: Unit ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112.service not loaded. 2024-02-13T03:39:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-13T03:39:15.898 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi112" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112' 2024-02-13T03:39:15.899 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-13T03:39:15.900 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/ubuntu@smithi080.front.sepia.ceph.com/crash 2024-02-13T03:39:15.901 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/crash -- . 2024-02-13T03:39:15.942 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/ubuntu@smithi112.front.sepia.ceph.com/crash 2024-02-13T03:39:15.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/crash -- . 2024-02-13T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2024-02-13T03:39:15.991 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-13T03:39:15.991 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-13T03:39:16.023 INFO:tasks.cephadm:Compressing logs... 2024-02-13T03:39:16.023 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T03:39:16.065 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T03:39:16.092 INFO:teuthology.orchestra.run.smithi080.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-13T03:39:16.092 INFO:teuthology.orchestra.run.smithi080.stderr:: No such file or directory 2024-02-13T03:39:16.092 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T03:39:16.092 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log: /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mon.smithi080.log 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr: 88.4% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.log 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr: 88.8% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mgr.smithi080.vyzlfg.log 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mon.smithi080.log: 90.3% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T03:39:16.093 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.log: /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mgr.smithi080.vyzlfg.log: 88.6% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T03:39:16.095 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-13T03:39:16.095 INFO:teuthology.orchestra.run.smithi112.stderr:: No such file or directory 2024-02-13T03:39:16.096 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T03:39:16.096 INFO:teuthology.orchestra.run.smithi080.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T03:39:16.096 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log: 88.5% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T03:39:16.097 INFO:teuthology.orchestra.run.smithi112.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T03:39:16.099 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-02-13T03:39:16.099 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m0.015s 2024-02-13T03:39:16.099 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m0.015s 2024-02-13T03:39:16.099 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.009s 2024-02-13T03:39:16.190 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mgr.smithi080.vyzlfg.log: file size changed while zipping 2024-02-13T03:39:16.190 INFO:teuthology.orchestra.run.smithi080.stderr: 92.7% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mgr.smithi080.vyzlfg.log.gz 2024-02-13T03:39:16.264 INFO:teuthology.orchestra.run.smithi080.stderr: 90.7% -- replaced with /var/log/ceph/84391b2a-ca1f-11ee-95b9-87774f69a715/ceph-mon.smithi080.log.gz 2024-02-13T03:39:16.266 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-02-13T03:39:16.266 INFO:teuthology.orchestra.run.smithi080.stderr:real 0m0.185s 2024-02-13T03:39:16.266 INFO:teuthology.orchestra.run.smithi080.stderr:user 0m0.272s 2024-02-13T03:39:16.266 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m0.027s 2024-02-13T03:39:16.266 INFO:tasks.cephadm:Archiving logs... 2024-02-13T03:39:16.266 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/ubuntu@smithi080.front.sepia.ceph.com/log 2024-02-13T03:39:16.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T03:39:16.410 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/ubuntu@smithi112.front.sepia.ceph.com/log 2024-02-13T03:39:16.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T03:39:16.446 INFO:tasks.cephadm:Removing cluster... 2024-02-13T03:39:16.446 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 --force 2024-02-13T03:39:19.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84391b2a-ca1f-11ee-95b9-87774f69a715 --force 2024-02-13T03:41:57.604 INFO:tasks.cephadm:Removing cephadm ... 2024-02-13T03:41:57.605 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T03:41:57.628 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T03:41:57.646 INFO:tasks.cephadm:Teardown complete 2024-02-13T03:41:57.647 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112' 2024-02-13T03:41:57.876 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9be6a2f396e74e72b45e242452587dcf Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_84e714279aa684636ac645daf0a1a85c2094cfb3/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112' 2024-02-13T03:41:57.883 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-13T03:41:57.895 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-13T03:41:57.919 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-13T03:41:57.919 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-13T03:41:57.935 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-13T03:41:58.014 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-13T03:41:58.022 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-13T03:41:58.023 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T03:41:58.025 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T03:41:58.042 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2024-02-13T03:41:58.043 INFO:teuthology.orchestra.run.smithi080.stderr:bash: ntpq: command not found 2024-02-13T03:41:58.046 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T03:41:58.046 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-02-13T03:41:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 -209us[ -202us] +/- 81ms 2024-02-13T03:41:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -751us[ -744us] +/- 57ms 2024-02-13T03:41:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +1047us[+1054us] +/- 68ms 2024-02-13T03:41:58.047 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:41:58.048 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T03:41:58.048 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-02-13T03:41:58.048 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 -230us[ -230us] +/- 81ms 2024-02-13T03:41:58.048 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -738us[ -728us] +/- 57ms 2024-02-13T03:41:58.049 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1052us[+1062us] +/- 68ms 2024-02-13T03:41:58.049 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T03:41:58.050 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-13T03:41:58.059 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-13T03:41:58.060 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-13T03:41:58.071 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T03:41:58.160 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T03:41:58.223 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T03:41:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:41:58.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T03:41:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:41:58.286 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-13T03:41:58.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-13T03:41:58.309 INFO:teuthology.task.internal:Duration was 1562.032989 seconds 2024-02-13T03:41:58.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-13T03:41:58.317 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-13T03:41:58.317 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T03:41:58.320 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T03:41:58.367 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T03:41:58.372 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T03:41:58.720 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-13T03:41:58.720 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-02-13T03:41:58.721 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T03:41:58.745 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-02-13T03:41:58.746 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T03:41:58.767 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-13T03:41:58.768 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T03:41:58.788 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T03:41:58.857 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-13T03:41:58.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T03:41:58.859 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T03:41:58.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-13T03:41:58.960 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-13T03:41:58.961 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T03:41:58.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T03:41:59.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-13T03:41:59.024 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T03:41:59.030 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T03:41:59.054 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-02-13T03:41:59.081 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-02-13T03:41:59.109 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T03:41:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:41:59.129 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T03:41:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T03:41:59.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-13T03:41:59.176 INFO:teuthology.task.internal:Transferring archived files... 2024-02-13T03:41:59.177 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/smithi080 2024-02-13T03:41:59.178 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T03:41:59.218 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185/remote/smithi112 2024-02-13T03:41:59.219 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T03:41:59.258 INFO:teuthology.task.internal:Removing archive directory... 2024-02-13T03:41:59.258 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T03:41:59.260 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T03:41:59.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-13T03:41:59.320 INFO:teuthology.task.internal:Not uploading archives. 2024-02-13T03:41:59.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-13T03:41:59.329 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-13T03:41:59.329 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T03:41:59.331 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T03:41:59.346 INFO:teuthology.orchestra.run.smithi080.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 03:41 /home/ubuntu/cephtest 2024-02-13T03:41:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 03:41 /home/ubuntu/cephtest 2024-02-13T03:41:59.368 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-13T03:41:59.379 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-13T03:41:59.399 INFO:teuthology.nuke:Checking targets against current locks 2024-02-13T03:41:59.425 DEBUG:teuthology.nuke:shortname: smithi080 2024-02-13T03:41:59.425 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T03:41:59.449 DEBUG:teuthology.nuke:shortname: smithi112 2024-02-13T03:41:59.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T03:41:59.452 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 03:07:41.733154', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIs9gF0uJF7l4uk+bpLQcy2Tc+w2O03dLVx4pZs5u2B7ZrhFQQFpn0FjaFLEU+C1UxjL+WRs9ylCdLmwCO3LF4Y='} 2024-02-13T03:41:59.476 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-12_21:56:27-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7557185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 03:07:41.731401', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlhBeoPgnC/NZRGMfUuPLhAZfCfuZax//FmjalFq1XcVkxpCm+y4BCqsJXckUbkU6J/xEg/V5oKW6jdYQ0iP3c='} 2024-02-13T03:41:59.507 INFO:teuthology.orchestra.console.smithi080:Power off 2024-02-13T03:41:59.507 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T03:41:59.531 DEBUG:teuthology.orchestra.console.smithi080:power off output: Chassis Power Control: Down/Off 2024-02-13T03:41:59.531 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:41:59.546 INFO:teuthology.orchestra.console.smithi112:Power off 2024-02-13T03:41:59.546 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T03:41:59.556 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-02-13T03:41:59.570 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2024-02-13T03:41:59.571 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:41:59.598 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-02-13T03:42:03.557 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:42:03.598 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:42:03.667 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-02-13T03:42:03.709 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-02-13T03:42:07.673 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:42:07.711 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:42:07.783 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is off 2024-02-13T03:42:07.821 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-02-13T03:42:07.885 INFO:teuthology.orchestra.console.smithi080:Power off completed 2024-02-13T03:42:11.826 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T03:42:11.935 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2024-02-13T03:42:12.036 INFO:teuthology.orchestra.console.smithi112:Power off completed 2024-02-13T03:42:12.141 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1562.0329887866974 failure_reason: 'Command failed on smithi112 with status 5: ''sudo systemctl stop ceph-84391b2a-ca1f-11ee-95b9-87774f69a715@mon.smithi112''' flavor: default owner: scheduled_lflores@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9be6a2f396e74e72b45e242452587dcf status: fail success: false 2024-02-13T03:42:12.142 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T03:42:12.217 INFO:teuthology.run:FAIL